The differential geometry of perceptual similarity Antonio ...rhg/pubs/GrangerRGPEGa1.pdf ·...

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

ThedifferentialgeometryofperceptualsimilarityAntonioMRodriguez1andRichardGranger1*1BrainEngineeringLab,DartmouthCollege,HanoverNH03755,USA*Richard.Granger@gmail.com

RodriguezA,GrangerR(2017)Thedifferentialgeometryofperceptualsimilarity.BrainEngineeringLaboratory,TechnicalReport2017.2.www.dartmouth.edu/~rhg/pubs/GrangerRGPEGa1.pdf

AbstractHumansimilarityjudgmentsareinconsistentwithEuclidean,Hamming,Mahalanobis,andthemajorityofmeasuresusedintheextensiveliteraturesonsimilarityanddissimilarity.Fromintrinsicpropertiesofbraincircuitry,wederiveprinciplesofperceptualmetrics,showingtheirconformancetoRiemanniangeometry.Asademonstrationoftheirutility,theperceptualmetricsareshowntooutperformJPEGcompression.Unlikemachine-learningapproaches,theoutperformanceusesnostatistics,andnolearning.Beyondtheincidentalapplicationtocompression,themetricsofferbroadexplanatoryaccountsofempiricalperceptualfindingssuchasTversky’striangleinequalityviolations(1,2),contradictoryhumanjudgmentsofidenticalstimulisuchasspeechsounds,andabroadrangeofotherphenomenaonperceptsandconceptsthatmayinitiallyappearunrelated.Thefindingsconstituteasetoffundamentalprinciplesunderlyingperceptualsimilarity.

Introduction:ThefundamentalsofperceptualsimilarityWhendoimageslookalike?AllstandardEuclidean(andHamming,andMahalanobis,andalmostallother)standardmeasuresofsimilarityturnouttobeatoddswithhumansimilarityjudgments.Wespelloutwhythisisthecase,giveexplanatoryprinciples,andprovideanillustrativeapplicationtothewidely-usedJPEGcompressionalgorithm:JPEGhasbeenoutperformedviaextensivelearningbyneuralnetworkandMLmethods,whereasweoutperformitwithnostatistics,andnotraining.WeshowthattheJPEGfindingsfalloutasaspecialcaseoftheunderlyingbroadprinciplesintroducedhere,whichareapplicabletoawiderangeofunsupervisedmethodsthatentailsimilaritymeasures.Euclideanvectors’componentsareorthogonal,andthus

!a = (10000) and !b = (00010) are

equidistantfrom !c = (00001) :distances

!a!c and !b!c bothhaveHammingdistancesof2,

andEuclideandistancesof 2 .However,consideredasphysicalimages,theright-handpositioningofthe“1”valuesinvectors

!b and

!c renderthemmorevisuallysimilartoeachotherthaneitheristo

!a .Whensuch“neighbor”relationswithinavectorareconsidered,thenvectoraxesarenotorthogonal,andnon-Euclideanmetricscanreadilyyieldsmallerdistancesbetween

!b and

!c thanbetween !a and

!c .Humanjudgmentsofsimilarityimplyaparticulargeometricsystem,andasintheabovesimpleexample,itiseasytoshowthathumansimilarityjudgmentsdonotconformtoEuclidean,Hamming,Mahalanobis,ortheothermostcommonlyusedsimilaritymetrics;rather,wewillshowthattheyconformtoRiemanniangeometry.

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page2of23

Moreover,humansimilarityjudgmentsarenotsolelyafunctionofthestimulithemselves;theydependalsoontheoperationsinternallycarriedoutbytheperceiver.Givenphysicalstimuli(e.g.,theEnglishspeechsounds/ra/and/la/)aredistinguishabletosomeperceivers(nativeEnglishspeakers)butdifficulttodiscriminatebyotherperceivers(e.g.,nativeJapanesespeakers),asaresultofpriorexperienceoftheperceiver(3-7).Thus“re-coding”thestimulidoesnotcontributetoasolution(8-10).Rather,wewishtoidentifytheperceptualmetricsthatagivenperceiveruseswhenjudgingsimilarityanddifference.Asinthevector“neighbor”example,perceptualbrainsystemanatomyreflectsnon-Euclideanmetrics:thesignaltransmittedfromonegroupofneuronstoanotherdirectlycorrespondstomappingsamongnon-Euclideanspaces.Wederiveaformalismfromsynapticconnectivitypatternsandshowthatthesystemmatchesandexplainsindividualhumanempiricaljudgments.Weapplythederivedsystemtothewell-studiedJPEGcompressiontask(solelytodemonstratethereal-worldefficacyofthepresentedprinciples).Conveniently,severalstatisticalmachine-learningapproacheshaverecentlybeenshowntooutperformthesturdyhand-constructedJPEGmethod,viaextensivetrainingonimagedata(11,12).Thoseresultsmaybeviewedascallingforpotentialexplanatoryprinciplesthatmayunderlietheirsuccesses.Whatstructureinthedataisbeingstatisticallyidentifiedbytheselearningapproaches?ThemethodderivedinthepresentmanuscriptiseasilyshowncapableofoutperformingJPEGaswell,withnoincreaseincomputationalcostoverJPEG,andusingnostatisticsandnotraining.TheresultsarisefromnewlypositedprinciplesthatunderlienotjustJPEG,butperceptualsimilarityingeneral;JPEGisshowntofalloutasaspecialcaseofthemethod.BeyondtheillustrativeJPEGexample,themetricsareshowntoprofferexplanatoryaccountsofarangeofempiricalperceptualfindings,notablyTversky’striangleinequalityviolations(1,2),contradictoryhumanjudgmentsofstimuliinothermodalitiessuchasspeechsounds,andbeyondsimpleperceptstoabstractconceptsandcategoriesthatmayinitiallyappearunrelated.ResultsWhatJPEGdoes,andaprinciplederrorAnylossycompressionalgorithmtradesoffimagequalityforimageentropy:howtheimageappearsvs.howmuchspaceitcanbestoredin.TheJPEGstandard(13,14)transformsanimageintoafrequencybasis,andencodeseachofthefrequencycomponentswithadifferentamountofprecision(tendingtoencodelow-frequencycomponentswithmoreprecisionthanhigh-frequencycomponents),thusselectivelyintroducingmodesterrorspreferentiallyintohighfrequencycomponents,yieldinganewimagewithmoreerrorbutlessentropythantheoriginal;i.e.,animagejudged(viahumanviewers)tobeoflesserquality,butcapableoffittingintoasmallerfilesize.AnaxiomaticerrorincorporatedintoJPEG(andindeedintomostcompressionmethods,andmostmeasuresofsimilarityanddissimilarity),istheassumptionthatthefrequencybasisvectorsareorthogonal,andthusthatchangestoanyoneofthemdonotimpactthe

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page3of23

others.InEuclideanspace,thesebasesareindeedorthogonal;inRiemannianspacetheyarenot.Weshowthathumanperceptualsimilarityjudgmentsareconsistentwithnon-orthogonalbases,properlytreatedasaRiemannianspace,notEuclideanoraffine.ForJPEG,let

!p bea64-dimensionalvectorcomprisedofintensityinformationfroman8x8blockofpixels(weinitiallyfocusonmonochromeimagesforexpositorysimplicity).Wedefineamatrix Fx,y whoseelementsaretheintensitiesofthepixelsatlocations (x, y) .JPEGencodingproceedsviathefollowingfoursteps:1)Centerimageintensitiesaround0:

′Fx,y = Fx,y −128 .

2)RepresentimageaslinearcombinationoffrequencycomponentsThediscretecosinetransform(DCT)operatorTisgivenby:

Tu,v = 14α (u)α (v) ′Fx,y cos

y=0

N−1

∑ (2x +1)uπ2N

⎡

⎣⎢

⎤

⎦⎥

x=0

N−1

∑ cos(2y +1)vπ

2N⎡

⎣⎢

⎤

⎦⎥ (2.1)

where

α (x) =1

2

1

if x = 0else

⎧⎨⎪

⎩⎪andwhereN=8forJPEG.

(TheDCTisalinearoperatoractingonthe(centered)image ′F ;i.e., T ( ′F ) = T i ′F )3)QuantizationandroundingoffrequencycomponentsEachofthe64dimensionsofTareindependentlyscaledbyapredetermined“quantization”matrix QC (withtheintendedeffectofdiscardingless-relevantinformationinthedata),withdifferentmatricesdefinedfordifferent“calibers”Coftheerror/entropytradeoff.TheoperatorRsimplydividestheelementsofatransformedinput(T)bythedesignatedquantizationmatrix QC :

Ru,v = Tu,v QCu,v or,inmatrix

notation, R = QC−1 iT

Fullquantizationofinput ′F ,then,isaccomplishedinJPEGby

Z = round(R i ′F ) = round(QC−1 iT i ′F ) (2.2)

TheJPEGstandardestablishespreëstablishedquantizationmatrices (QC ) foranygivendesiredcompressionfactorC,i.e.,foragivenreductioninqualityandcommensuratedecreaseinentropy.Thesequantizationmatriceswereconstructedbyhand(13,14),withnonon-manualmethodforarrivingatitsvalues(15).

4)EntropyencodingThenumericalelementsofJPEG’squantizedandroundedimageZareencodedviaHuffmancoding,suchthatthemostfrequentlyusednumericalvaluesareassignedtheshortestbitrepresentation,thustakingadvantageofthereducedentropyofthequantizedinput,toenablethecompressedimagetobestoredinafilesmallerthantheoriginal.AlthoughHuffmancodingisusedbyJPEG,anynumberofentropyencodingmethods(suchasarithmeticencoding)wouldsuffice.

WefocusonaspecificerroneousassumptionunderlyingJPEG(andotherperceptualcompressionmethods):theuseofEuclideanmeasuresofsimilarity.Infact,anygivenpixel

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page4of23

pisnotperceptuallyindependentofnororthogonaltoneighboringpixels.Asintheexamplesintheintroductionsection,neighboringpixelsareperceivedbyaviewerasbeing“closer”toeachotherthanaremoredistalpixels,andthisaltersperceivedsimilarity.Correspondingly,the64frequencybasesintheDCTarenotperceptuallyorthogonal(thoughtheyareorthogonalinEuclideanspace):someareperceptualjudgedmoresimilartoeachotherthanothersbyhumanviewers.Moreover,thesejudgmentsaredependentontheircoefficients,andthushavedifferentsimilaritiesindifferentpartsoftheDCTbasisspace.Thecurvatureofthespacethusisnotaffine,butratherRiemannian(Figure1).WeintroduceanappropriateRiemanniantreatmentofperceptualsimilarity.WeshowthattheresultingmethodcanreadilyoutperformJPEG,butmoreimportantly,ithasexplanatorypower:JPEGemergesasaspecialcaseofthegeneralmethod,andtheunderlyinggeometricprinciplesofhumanperceptionbecomemorecloselyexplained.TheRiemanniangeometricprinciplesofperceptionThreegeometricspacesAssumeanimageof64pixels(grayscale,fortemporarypedagogicalsimplicity),arrangedasan8x8array.InJPEGandallotherstandardcompressionmechanisms,theimageistreatedasanarbitrarily,butconsistently,ordered64-dimensionalvector,suchthateachvectorentrycorrespondstotheintensityatoneofthe64pixelsinthe8x8array.Thisrendersthedataintosimplevectorformat,enablingtheapplicabilityofvectorandmatrixoperations.However,itdoessoatthecostofeliminatingtheneighborrelationsamongpixelsinthephysicalspace.(Typically,the64pixelsareordered(arbitrarily)withentries1-8fromthetoprowofthearray,9-16fromthenextrow,andsoon.)Theeliminationofneighborrelationswouldbeirrelevantifhumanperceptionofapixelweremodulatedequally,ornotatall,bythecharacteristicsofneighboringanddistantpixelsalike;thisturnsoutnottobethecase.WeforwardthealternativeinwhichtheimageisdescribedintermsofaphysicalspaceΦ with3dimensions(forxandylocations,andintensity),foreachofthe64pixels.Thiscorrespondstoatransformationofthe“feature”space F intophysicalspaceΦ .Theexplicitrepresentationofphysicalpixelpositionsenablesperceptualencodingsthatusepixelpositionasaparameter.Inadditiontofeaturespace F andphysicalspaceΦ ,weintroduceathirdspace,Ψ ,whichweterm“perceptualspace,”thatincludesrepresentationofperceptualgeometricrelationsamongtheimageelements(Figure2).Transformsintothisperceptualspace,accomplishedbydifferentialgeometry,willbeshowntodirectlycorrespondtohumanperceptualsimilarityjudgments.BrainconnectomesareRiemannianFigure2showssampleanatomicalconnectivityamongbrainregions,anditsformalproperties.Figure2ashowsaninstanceoftypicalmammalianthalamocorticalandcortico-corticalsynapticprojections(16-20).Theprojectionpatternfromonecellularassemblytoanotherisnotperfectly“pointtopoint”(i.e.,eachcellprojectingtoexactlyonetopographicallycorrespondingtargetcell)norcompletelydiffuse(withnotopography);rather,theprojectionis“radiallyextended,”suchthateachelementcontactsarangeoftargetsroughlywithinaspatialneighborhoodorradiusaroundatarget.Figure2bshowsasimplevectorencodingoftheseprojectionpatternswithcorrespondingsynapticweights

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page5of23

′n , ′′n , etc.Figure2cshowsexamplesoftypicalphysiologicalneuralresponsesinearlysensorycorticalareasthatcanarisefromtheseconnectivitypatterns.Figure2dcontainsthegeneralformofaJacobianmatrixdenotingtheoveralleffectofactivityintheneuronsofaninputarea x ontheneuronsintargetarea f ;eachentryintheJacobiandesignatesthechangeinanelementof f asaconsequenceofagivenchangeinanelementof x .Figure2eisanexampleinstanceofsuchaJacobian,correspondingtothesynapticconnectivitypatterninFigure2b.Intuitively,aJacobianencodestheinteractionsamongstimuluscomponents.Ifavectorcontainedpurelyindependententries(asinanimaginedperfectlypoint-to-pointtopographywithnolateralfan-inorfan-outprojections),theJacobianwouldbetheidentitymatrix:onesalongthediagonalandallotherentrieszeros.Eachvectordimensionthenhasnoeffectonotherdimensions:agiveninputunitaffectsonlyasingletargetunit,andnoothers.Inactualconnectivity,whichdoescontainsomeradiallyextendedprojections,thereareoff-diagonalnon-zerovaluesintheJacobiancorrespondingtotheslightlynon-topographicsynapticcontacts(Figure2b).Whentheinputandoutputpatternsaretreatedasvectors,anyoff-diagonalJacobianelementsreflectinfluencesofonedimensiononothers:thedimensionsarenotorthogonal,andthevectorsareRiemannian,notEuclidean(21).Allperceptualsystemscanbeseentointrinsicallyexpressa“stance”onthegeometricrelationsthatoccuramongthecomponentsofthestimuliprocessedbythesystem.Inthedegeneratecaseofnooff-diagonalelements,thesystemwouldactasthoughitassumesindependenceofcomponents(Euclideanvectors).Inallpathwayscharacteristicofmostthalamo-corticalandcortico-corticalprojections,however,theprocessinginherentlyassumesnon-Euclideanneighborrelationsamongthestimuli.ItisnotablethatanybankofneuronalelementswithreceptivefieldsconsistingeitherofGaussiansoroffirstorsecondderivativesofGaussians,willhavepreciselytheeffectofcomputingthederivativesoftheinputsinjusttheformthatarisesinaJacobian(seeequation(2.3)below)(22-26).PhysiologicalneuronresponsepatternsthusappearthoroughlysuitedtoproducingtransformsintospaceswithRiemanniancurvatures.(Notably,thisimpliesthatasynapticchange(e.g.,LTP)causesspecificre-shapingofneurons’receptivefields,modifyingthecurvatureofthespaceofthetargetcellsinagivenprojectionpathway.)ThematrixJinFigure2ddescribestheparticulartransformfromaninputspacetoanoutputspace.Thisisaninstanceofspecifyingthedifferencesbetweenaperceptualinput,versusaperceptthatisreceivedviathisprojectionpathway.AperceiverwillprocessaninputasthoughitcontainstheneighborrelationsspecifiedbytheJacobian.ThemapfromphysicaltoperceptualspaceNeighboringentriesinavector,likeadjacentnotesonapianokeyboard,areclosertoeachotherthanentriesfromother,non-neighboringdimensions.Thefeaturesthusdonotconstituteindependentdimensions(or,putdifferently,thedimensionsarenotorthogonal).Inthese(extremelycommon)cases,targetperceptualdistancesarecorrectlyrenderedbyRiemannianratherthanbyEuclideanmeasures.Itisnotablethatthisnotanexceptionbut

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page6of23

thenormalcaseforperception.Euclideanvectorsdonottreatconstituentsashavingneighbors,butperceiversdo.Wewishtodetermine,then,howchangestoanimagewillbeperceived.Achangeinphysicalspace(i.e.,theimage)canbedirectlymeasured.Thecorrespondingpredictedchangeinperceptualspacecanthenbecomputedviaametrictensorwhichmeasuresdistanceintheperceptualspacewithrespecttopositionsinphysicalspace.ThismetrictensoriscomputedviaaJacobianthatmapsfromdistancesinphysicalspacetodistancesinperceptualspace(Figure3).Forthemap

µF →Φ theJacobian JF →Φ willbea3x64

matrix:

JF →Φ =

dΦ0df0

dΦ0df1

dΦ0df2

!dΦ0df63

dΦ1df0

dΦ1df1

dΦ1df2

!dΦ1df63

dΦ2df0

dΦ2df1

dΦ2df2

!dΦ2df63

⎡

⎣

⎢⎢⎢⎢⎢⎢⎢⎢⎢

⎤

⎦

⎥⎥⎥⎥⎥⎥⎥⎥⎥

(2.3)

(Supplementalsections§2.5-§2.11givesamplevaluesusedtogeneratespecificJacobiansforimageprocessing,asintheexamplesshowninFigure4).ThisJacobianenablesidentificationofadistancemetricforthefeaturespace F withrespecttoitsembeddinginphysicalspaceΦ intermsofthemetrictensor g (seeSupplementalsection§2.5forexamplesofvaluesused):

gΦ:F ( !x) = JF →ΦT ( !x) i JF →Φ( !x) (2.4)

i.e.,themetrictensorgisusingmeasuresinspaceΦ appliedtoobjectsinspace F ,or,putdifferently,thetensormeasuresdistancesin F withrespecttomeasuresinspaceΦ .Then,mappingphysicalspaceΦ toperceptualspaceΨ (viaJacobianoperator JΦ→Ψ )defineshowfeaturesinthephysicalspaceareperceivedbyaviewer,enablingaformaldescriptionofhowchangesinthephysicalimageareregisteredasperceptualchanges.SpecificconstructionoftheJacobianmapping JΦ→Ψ Informationfromthephysicalstimulusorfromtheperceiver(orboth)enablesconstructionofaJacobiantomapfromphysicalvectorstotheperceptualspaceaperceivermayuse.SuchaJacobian, JΦ→Ψ ,canbeobtaineddirectlyfromeithersynapticconnectivitypatternsorfrompsychophysics–eitherbyaprioriassumptionsorfromempiricalmeasurements.(i) SynapticJacobian:

a) EmpiricalMeasureanatomicalconnectionsandsynapticstrengths,ifknown;theJacobianisdirectlyobtainedfromthosedataasinFigure2.Thesemeasurestypicallyareunavailable,butaswillbeseen,approximationsmaybedrawnfromasetofsimpleconnectivityassumptions.

b) Estimated

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page7of23

Assumeradiusofprojectionfan-outfromacorticalregiontoatargetregion(Figure2a),basedonmeasuresoftypicalsuchprojectionsintheliterature(16-20),andestimateafactorbywhichdistancesamongstimulusinputfeatures(e.g.,pixels)influencerelatednessofthefeatures,andresultingcurvatureoftheRiemannianspaceinwhichtheyarethusassumedtobeperceptuallyembedded.

(ii) PsychophysicalJacobian:a) Empirical

Measureconstituentphysicalfeaturesofthestimuliandcalculatedistancesamongstimulusfeatures,suchaspixelsize,pixeldisparity,viewingdistance,andobtainempiricalmeasuresofhuman-reporteddistances;theJacobianisthesetofrelationsamongpsychologicalandphysicaldistancesasinEquation(2.3).

b) EstimatedAssumeGaussianfall-offofrelatednessofneighboringpixelsinastimulus;measureconstituentfeaturesasin(iia)andestimateafactorbywhichdistancesbetweenstimulusinputfeatures(e.g.,pixeldistancesinxandydirections)influencetherelatednessofthefeatures(andtheresultingcurvatureoftheRiemannianspaceinwhichtheyareassumedtobeperceptuallyembedded).

Inthepresentworkweproceedwithmethod(iib),i.e.,measuring(Euclidean)physicaldistancesamongpairsofinputsandpositingarangeofcandidatefactorsbywhichthephysicaldisparityamongfeaturesmaygiverisetoperceivedfeatureinteractions.Weshowaseriesofresultingfindingscorrespondingtothisrangeofdifferenthypothesizedfactors(Supplementalsection§2.5).HavingobtainedaJacobianbyanyoftheabovemeans,wecomputemetrictensorgasinequation(2.4).(Thetensoralternatelymaybeobtainedincondition(ii)usingthe

covariancematrixΣ frompsychophysicalexperimentaldata: gΨ:Φ = ΣΨ:Φ−1 .)(See

supplementalsection§1.4).Wemaymovetheobtainedmetric gΨ:Φ fromphysicalspacetofeaturespace,obtaininga

newmetric gΨ:F thatmeasuresdistancesinthefeaturespacewithrespecttotheperceptualspace:

gΨ:F = JF →ΦT i gΨ:Φ i JF →Φ (2.5)

ThisnewmetricinthefeaturespacenowcomputestheRiemanniandistancesamongdimensionsthatholdinthefeaturespace.Themetriccanbeusedtocomputethematrixofalldistancesamongallpairsoffeatures

xi ,x j inacolumnvector

!x ofdimensionalityk:

dist(xi ,x j ) = (2π )k gΨ:F( )−1 2

exp − 12

(xi − x j )T gΨ:F (xi − x j )( ) (2.6)

(where !a isthedeterminantof

!a ).(SampledistancematricesforselectedspecificmeasuredvisualparametersareshowninSupplementalsection§2.7;tables§2-§4).

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page8of23

Thedimensionsofafeaturevectorin(Euclidean)space F areorthogonal,butthedimensionsofthecorrespondingvectorin(Riemannian)perceptualspaceΨ arenotorthogonal;rather,thepairwisedistancesamongthedimensionsaredescribedbyEq.(2.6).MethodsDerivationofRiemanniangeometricJPEG(RGPEG)JPEGmodifiestheimagefeaturevector,introducingerror(thedistancebetweentheoriginalandmodifiedvector),suchthatthemodifiedvectorhaslowerentropy,andthuscanbestoredwithasmallerdescription.Correspondingly,wetoowillmodifythefeaturevector,introducingerrorinordertolowerentropy,butinthiscaseusinggraph-basedoperationsonnon-Euclideandimensiondistances(Eq.(2.6)).WeintroducetheRiemanniangeometricperceptualencodinggraph(RGPEG)method.JPEGusesa(hand-constructed)quantization(“Q”)matrixthatspecifiestheamountbywhicheachofthe64DCTdimensionswillbeperturbed,suchthatwhentheyaresubjectedtointegerrounding,theywillexhibitlowerentropy.WereplacetheJPEGquantizationoperationswithaprincipledformulathatcomputesperturbationsofbasisdimensionstoachieveadesiredentropyreductionandcommensurateerror–butinperceptualspaceratherthaninfeaturespace.Specifically,thesurrogatequantizationstepmovestheimageinperceptualspacealongthegradientoftheeigenvectorsoftheHamiltonianofthebasisspace.WeshowthattheresultingcomputationcanoutperformJPEGoperations(oranyoperationsthattakeplaceinfeaturespaceratherthaninperceptualspace).DerivationofentropyconstraintequationWedefineagraphwhosenodesarethe64basisdimensionsofthefeaturespace.(ForJPEGthisbasisisthesetof642-ddiscretecosinetransforms;forRGPEGwederivethegeneralizationofthisbasisforperceptualspace,showingtheDCTtobeaspecialcase).Activationpatternsinthegraphcanbethoughtofasthestateofthespace,andoperationsonthegrapharestatetransitions.Wedefine Ω(x,s) asthestatedescribingtheintensityofeachofthepixelsinthe(8x8)image,suchthat Ω(x,0) istheoriginalimage,andany Ω(x,s) fornon-zerosvaluesisanalteredimage,includingthepossiblecompressedversionsoftheimage.Wedefinethesvaluestobeinunitsofbitsxlength;correspondingtothenumberofbitsrequiredtostoreagivenimage,andthuscommensuratewithentropy(seeSupplementalsection§2.3).Wewishtoknowhowtochangetheimagesuchthattheentropywillbereduced.Changestotheimagewithrespecttoentropyareexpressedas

∂Ω(x,s)∂s

Wetreattheproblemofsuchimagealterationsintermsoftheheatequation(see,e.g.,(27,28),andseeSupplementalsection§2.3).Weequatethesecondderivativeoftheimagestatewithrespecttodistance,withthederivativeoftheimagewithrespecttoentropy:

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page9of23

∂Ω(x,s)∂s

=∂2Ω(x,s)

∂x 2 (2.7)

Wetermeq.(2.7)theentropyconstraintequation;wewanttoidentify Ψ(x,s) thatsatisfiesthisequation,suchthatwecangeneratemodificationsofanimagetoachieveanewimageexhibitingareductionentropy(andcorrespondinglyincreaseinerror).Viaseparationofvariablesweassumeasolutionoftheform Ω(x,s) =ω (x)φ(s) (2.8)wherethefunctionω isonlyintermsofpositioninformationxandthefunctionφ onlyintermsofentropys.Thustheformerconnotesthe“position”portionofthesolution,i.e.,valuesofimagepixelsregardlessofentropyvalues,whereasφ istheentropyportionofthesolution.Wecanformulatetwoordinarydifferentialequationscorrespondingtothetwosidesofthepartialdifferentialequationinequation(2.7):

∂Ω(x,s)∂s

=ω (x)dφ(s)

dsand

∂2Ω(x,s)

∂x 2=

d 2ω (x)

dx 2φ(s)

whichbothequalthesamevalueandcanthusbeequated:

ω (x)

dφ(s)ds

=d 2ω (x)

dx 2φ(s) (2.9)

whichcanbesimplified

1φ(s)

dφ(s)ds

=1

ω (x)d 2ω (x)

dx 2

Sincethetwofunctionsareequal,theyareequaltosomequantity(whichcannotbeafunctionofxors,sincetheequalitywouldthennotconsistentlyhold).Wecallthatquantityλ .Therecanbeadistinctλ valueforeachcandidatesolutioni.Foranysuchgivensolution,theentropytermis:

dφ(s)ds

= φ i(s)λ i

whosesolutionis

φ i(s) = eλ is (2.10)Asmentioned,therewillbeisolutionsforeachvalueofλ .(Seesupplementalsection§2.3).Forthepositionterm:

d 2ω (x)

dx 2=ω i(x)λ i (2.11)

thesolutionisintheformoftheFourierdecomposition

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page10of23

ω i(x) = ci

!γ i(x)i∑ (2.12)

wherethe !γ i termsaretheeigenvectorsoftheLaplacianofthepositionterm,eq.(2.11),

andwherethe ci termscorrespondtothecoefficientsoftheeigenvectorbasisoftheinitialconditionofthestate Ω(x,s) correspondingtotheinitialimageitself, Ω(x,0) .(Preciseformulationofthe ci isshowninthenextsection).The64solutionsoftheFourierdecompositionformthebasisspaceintowhichtheimagewillbeprojected.(ForJPEG,thisisthediscretecosinetransformorDCTset,asmentioned;wewillseethatthiscorrespondstoonespecialcaseofthesolution,foraspecificsetofvaluesoftheentropyconstraintequation.)ApplicationofentropyconstraintequationtoimagefeaturespaceConsiderthegraph(Figure4a)whosenodesaredimensionsoffeaturespace F andwhoseedgesarethepairwiseRiemanniandistancesbetweenthosedimensionsasdefinedbythedistancematrixofequation(2.6)insectionIIIc.Thedistancematrixcanbetreatedasthe

adjacencymatrix !A ofthatgraph.Wecomputethedegreematrix

!D via

Dii = Aij

j=1

n∑ for

!A withrowindices i = 1,…,m andcolumnindices j = 1,…,n .ThegraphLaplacianis

!Lg =

!D −!A ,andthenormalizedgraphLaplacianisthen L = D 1

2 Lg D 12 .

ThetotalenergyofthesystemcanbeexpressedintermsoftheHamiltonian H ,takingtheform H = L+ P whereListheLaplacianandP(correspondingtopotentialenergy)canbeneglectedasaconstantforthepresentcase;thehamiltonianisthusequivalentforthispurposetothelaplacian:

H =

∂2Ω(x,s)

∂x 2 (2.13)

Intuitively,theHamiltonianexpressesthetradeoffsamongdifferentpossiblestatesofthesystem(Figure4);appliedtoimages,theHamiltoniancanbemeasuredforitserrors(distancefromtheoriginal)ononehand,anditsentropyorcompactnessontheother:amorecompactstate(lowerentropy)willbelessexact(highererror),andviceversa.Theaimistoidentifyanoperatorthatbeginswithapointinfeaturespace(animage)andmovesittoanotherpointsuchthatthechangesinerrorandentropycanbedirectlymeasurednotinfeaturespacebutinperceptualspace(Fig3).Thusthedesiredoperatorwillmovetheimagefromitsinitialstate(withzero“error,”sinceitistheoriginalimage,andaninitialentropyvaluecorrespondingtotheinformationintheimagestate)toanewstatewithanewtradeoffbetweenthenow-increasederrorandcorrespondingentropydecrease.TheHamiltonianenablesformulationofsuchanoperator.TheeigenvectorsoftheHamiltonian(Figure4c)constituteacandidatebasissetfortheimagevector(Figure4d),andsince HΩ = λΩ ,theeigenvaluesλ oftheHamiltoniancanprovideanoperator U (s) correspondingtoanygivendesiredentropys(seeSupplementalsection§2.3).Aswewill

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page11of23

see,the φ i(s) functionisthedesiredupdateoperator,movingthepointcorrespondingtothestate(theimage)toahigher-errorlocationinperceptualspacetoachievethedecreasedentropylevelcorrespondingtos.Theseparatedcomponentsofthestateequation(2.8)formthepositionsolutionandentropysolutiontotheequation,respectively. Ω(x,s) =ω (x)φ(s) Thepositionportion, ω (x) ,wasshowninequation(2.12)tobe

ω i(x) = ci

!γ i(x)i∑

andtheentropyportion, φ(s) ,wasshowninequation(2.10)tobe

φ i(s) = eλ is Theformerexpressesthesetofpositionalconfigurationsforeachgivensolutionandthelatterprovidesthefoundationfortheupdateoperatorforstatesi,toachieveentropylevels,wheretheλ valuescorrespondtotheeigenvaluesoftheHamiltonian.Combiningtheterms,weobtain

Ω(x,s) =ω i(x)φ i(s) = ci

!γ i(x)!φ i(s)

i∑ (2.14)

Toputtheseoperationsinmatrixform,wedefinethematrix !Γ composedofthecolumn

vectors !γ i(x) ,i.e.,theeigenvectorsofequation(2.12).Wedefinethefinalformofthe

updateoperator, U (s) ,tobethematrixcomposedofcolumnvectors !φ i(s) .(Each

!φ i(s)

hasonlyasinglenon-zeroentry,inthevectorlocationindexedbyi,andthus U (s) isadiagonalmatrix).Thetransformationstepsforalteringanimagetoadegradedimagewithloweredentropyandincreasederror,then,beginswiththeimagevector (

!f ) ,andprojectsthatvectorinto

theperceptualspacedefinedbytheeigenvectorbasisfromequation(2.12),suchthat ′

!f = Γ ⋅

!f (2.15)

Thevector ′

!f formstheinitialconditionsoftheoriginalimage,transformedinto

perceptualspace(bythe !Γ matrix,composedofthe

!γ i eigenvectorsfromequation(2.12)asthecolumnsof

!Γ ).Thevalues f i of

!′f constitutethevaluesofthe ci coefficientsthat

willbeusedinequation(2.14).Havingtransformedthevectorintoperceptualspace,theupdateoperatoristhenapplied ′′

!f = U (s) ⋅

!′f = U (s) ⋅

!Γ ⋅!f (2.16)

Theinitialimagenowhasbeenmovedintoperceptualspace (

!f →

!′f ) ,andmovedwithin

thatspacetoapointcorrespondingtoentropylevels (!′f →!′′f ) ,withacorresponding

increaseinerror(whichwillbemeasured).

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page12of23

Thelower-entropyimage, ′′f ,thushasbeenscaledsuchthatitnowcanbeencodedviaroundingintoamorecompactversion:

!′′′f = round( ′′

!f ) = round(U (s) ⋅

!Γ ⋅!f ) (2.17)

Anysubsequentencodingstepmaythenbeapplied,suchasHuffmanorarithmeticcoding,operatingontheroundedresult.Theseareequivalentlyapplicabletoanyothermethod(JPEG,RGPEG,orother)ofarrivingatatransformedimage,andarethusirrelevanttothepresentformulation.Weinsteadfocusonthedirectmeasuresoferrorandofentropy.WeproceedtocomparethesemeasuresdirectlyforJPEGandforthenewlyintroducedRGPEG.UpdateoperatormovesimagetolowerentropystateandminimizeserrorincreaseTheimage

!f nowhasbeenmovedfromfeaturespacetotheperceptualspacedefinedby

theeigenvectorbasisofequation(2.12),asinFigure4c,selectingaqualitylevel(seeSupplementalsection§2.3),applyingtheappropriateupdateoperator,androunding,resultinginequation(2.17).AsdescribedinsectionIIId,thesecomputationsdependedonconstructionofaJacobianeitherviaknowledgeof(orestimatedapproximationof)theanatomicalpathsfrominputtopercept(synapticJacobian),orviaempiricalpsychophysicalmeasures(psychophysicalJacobian).WecarriedoutseveralinstancesofcomputedcompressionviaanestimatedsynapticJacobian,composedbymeasuringdistancesbetweenpixelsonascreenimage,measuringviewingdistancefromthescreen,convertingthesetoviewingangle,andmeasuringallpixelsintermsofviewinganglesandthedistancesamongthem(Supplementalsection§2.5,andsupplementaltable§1).ExamplesofcomputedHamiltoniansandeigenvectorbasesareshowninFigure4eand4gforaparticularempiricalpixelsizeandviewingdistance(Supplementalsection§2.5);theformulaeshowhowanyempiricallymeasuredfeaturesgiverisetoacorrespondingHamiltonian.AsetofseveraladditionalsampleHamiltoniansandeigenvectorbasesareshowninSupplementalfigures§9-§13.Insum,JPEGassumesitsbasisvectors(discretecosinetransforms)tobeorthogonal,whichtheyareinfeature(Euclidean)space,butnotinperceptual(Riemannian)space.Asshown,theperceptualnon-zerodistancesamongbasisdimensionscanbeeitherempiricallyascertainedviapsychophysicalsimilarityexperiments,asinthepsychophysical-jacobianmethod,orassumedonthebasisofpresumptivemeasuresofanatomicaldistances(orapproximationsthereof)asinthesynaptic-jacobianmethod,orcalculatedonthebasisofphysicallymeasureddistancesinthephysicalspace,asinthephysical-distance-jacobianmethod(seeSupplementalsection§2.5).Inthepresentpaperwehavepredominantlytestedtheestimatedpsychophysicaljacobianmethod(methodiibabove),which(perhapssurprisingly)isshown,byitself,tooutperformJPEG.Fromthesemethods,wederivedHamiltoniansfromtheimagespace,andeigenvectorbasesfromtheHamiltonians,andshowedthattheJPEGDCTbasiswasaspecialcasewithparticularsettingsshowninSupplementalfigure§12.SidebysidecomparisonofJPEG/RGPEG

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page13of23

Performingcompressionwithmultiplesetsofparameters(seeSupplementalFigures§15-§24)yieldedempiricalresultsenablingcomparisonsoftheerrorandentropymeasuresfortheJPEGmethodandthemethod(RGPEG)derivedfromtheRiemanniangeometricprinciplesdescribedherein.Wehaveshownthatforspecificassumptionsofgeometricdistanceandofperceivedintensitydifference,theJPEGmethodoccursasaspecialcaseofthegeneralRGPEGprinciples(Supplementalsection§2.11.1).Itisintriguingtonotethat,usingsimpleestimationsofgeometricdistanceandlogscaleintensitydifferences,thegeneralizedRGPEGmethodtypicallyoutperformstheJPEGspecialcase,asexpected;Figure5showsonesuchdetailedsidebysidecomparison;manymoreareshowninSupplementalfigures§15-§24).ItalsoisnotablethatthecomputationalspaceandtimecostsfortheRGPEGmethodareidenticaltothoseforJPEG(Supplementalsection§2.12).Figure5showsarangeofcompressedversionsofasampleimage(fromtheCaltech256dataset),alongwiththemeasuresoferror(er)andentropy(en)foreachimage.Themethodcanmostclearlybeseentoproducefewerartifactswhencomparedatrelativelyhighcompressionlevels(highentropyandhigherror);thesearecleartoqualitativevisualinspection;thefigurealsoshowsquantitativeplotsofthetradeoffsofvaluesamongerrorandentropyforasetofselectedqualitylevels.Acrossarangeofqualitysettings,theerrorandentropyvaluesforRGPEGoutperformthoseforJPEG.Discussion:derivationofprinciplesOfprimaryinterestisnotthefactthatJPEGcompressioncanreadilybeoutperformedbythegeneralizedRGPEGmethod;rather,thereasonfortheoutperformanceisthatRGPEGembodiesanovelsetofprinciplesofperceptualsimilarity,andthattheseprincipleshaveexplanatorypowerforthesetofperceptualphenomenadescribed(ofwhichJPEGcompressionisoneinstance).Webrieflydiscusstheseexplanatoryprinciples.Physicalstimulussimilarityisdistinctfromperceptualstimulussimilarity.Standarddistancemeasures(Euclidean,Mahalanobis,etc.)(29)donotmatchhumansimilarityanddissimilarityjudgments(e.g.,SectionIIIcabove).Toaddressthis,somestandardapproaches“re-code”thestimulitomoreaccuratelyreflecttypicalsubjects’reportedperceivedsimilarityordissimilarityamongstimuli(30,31).Yetdifferentindividualperceiverscandifferentlyregisterdissimilarityamongidenticalphysicalstimuli,suchastheincompatiblesimilarityjudgmentsofspeechsoundsbynativespeakersofdifferentlanguages(4,6).Thesolutionisnottore-codethestimuli,butrathertoseparatelyrepresentphysicalstimuli(e.g.,speechsounds)ononehand,andtheparticularperceptualmappingsofthosestimuliontheother,viaametricoperationthattransformsdistancesfromthereferenceframeofthephysicalstimulusspaceintodistancesinanygivenperceiver’sperceptualreferenceframe(SectionIIId).PerceptualdistancesareintrinsicallyRiemannian.Euclideanvectordistancesassumeorthogonalityofconstituentvectordimensions.Thiscouldintheoryholdbutitisingeneralnotthecaseforperceptualstimuli.Theconstituentdimensionsofavectordonotdistinguishbetween“nearby”or“distant”dimensions,buthumanperceptualjudgmentstypicallydo.Riemannianspacecanintuitivelybethoughtofashaving“curved”axes(relativetoatangentspace)suchthatsomeregionsofagivenaxisare“closer”tosomeaxesandfartherfromothers,quitedistinctfromEuclideanspace.ThetoolsfromdifferentialgeometrypresentedhereenablestimuliinEuclideanfeaturespacetobemappedtophysicalandperceptualspaces;weforwardtheprinciplethatthesemappings

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page14of23

underliejudgmentsofperceptualsimilarity.Thispaperfocusesonexamplesinvisualdomains;additionalextensionstoauditorystimuli,andtoabstractconceptcategorizationareseparatefindingsbeingpursued.Perceptualmappingsarisedirectlyfromanatomicalstructureandphysiologicaloperation.a)Aperceptualsystemcannot“neutrally”processstimuli;anysystemcontainsintrinsicassumptionsabouttherelationsthatoccuramongthecomponentsofanystimuli.AperceptualsystemconnectomeencodesaJacobianeitherwithorwithoutoff-diagonalentries,causingittotreatstimuluscomponents(e.g.,neighboringpixelsinanimage)asdependentorindependent,respectively,andthenatureofanyoff-diagonalentriesdeterminestheexactdependencyrelationsamongthecomponents,correspondingtothespecificcurvatureofthemetricperceptualspace).b)CorticalneuronreceptivefieldsareoftencharacterizedintermsofGaussians(22-25,32,33).SuchcomponentsproduceoutputsthatcomputethepartialderivativesoftheirinputsinjusttheformneededfortheJacobianandtensorcomputationspositedhere;i.e.,typicalneuralassembliesappeartailoredtocomputingtransformsintoRiemanniantargetspaces.Synapticplasticitychangesthecurvatureofperceptualspace.Re-shapingneurons’receptivefieldsviasynapticmodificationdirectlychangestheJacobianmappingandthecurvatureofthetargetspace.Everysynaptic“learningrule”correspondstoamechanismbywhichexistingmetrictransforms(arisingfromtheconnectome)aremodifiedinresponsetostimuli.Alllearningrulescanbecastintermsofchangingcurvatureoftheprojectionfrominputtoperceptualspace.Transformscanbecomputedfromobservedbehavior.ConnectomesarealmostentirelyunmappedinsufficientdetailtoconstructaJacobian,andinanyeventperceptualspacesareformedbyacombinationofsuccessivefeedforwardstagesaswellasfeedbacktop-downinfluences.Agivenperceiver’sperceptualspacemaynonethelessbeelicitedempiricallybypsychophysicalmeasures(sectionIIId).Machinelearningisbasedonthesamegeometricprinciples.Unsupervisedlearningrulescanreadilyeducestatisticaldistributioncharacteristicsofdata,andtypicallyarejudgedbymeasuressuchaswithin-categoryvs.between-categorydistances(34-36).Butthediscoveryofunsupervisedstructureisnotneutralwithrespecttometricspaces:inresponsetoagivensetofdata,differentrulescausedifferentchangestotheJacobian,discoveringdifferentstructureinthedata(illustratedinthespecialcaseofJPEGencoding,butbroadlyapplicabletolearningstructureindata).RecentneuralnetapproacheshaveidentifiedlearningmethodsthatcanoutperformJPEG;thepresentwork,bycontrast,outperformsJPEGwithnotrainingandnostatistics,byidentifyingpreviouslyunnotedfundamentalsofperceptualencodingthatunderliesimilarityjudgments.Furtherprinciplesarisefromstudyofperceptualtransforms.InthepsychophysicalJacobianmethod(SectionIIId),forinstance,perceptualdistancesarisefromtheminimumdistancewithinthetargetRiemannianspace,i.e.,thegeodesic.Itcouldhavebeenthecasethatotherdistancesmightinsteadhavebeeninvolved.WeforwardtheprinciplethatperceiveddistancesarepredictedbymeasuresofRiemannianminimumdistance.Otherunderlyingprinciplesmaysimilarlyemergefromfurtherstudy.Applicationtoawell-studiedperceptualanomaly.

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page15of23

Tverskyandcolleagues(1,2)showedthatperceivedsimilarityjudgmentsofsomeclassesofstimuliviolatedthetriangleinequality:eventhoughstimuliAandBmayphysicallysharemorefeaturesthanAandC,thelattermaybejudgedmoresimilarthantheformer.ThepresentstudiessuggestthatsubjectsintheseexperimentsareperceivingthestimuliinaRiemannianspace(Figure6),inwhichaseemingly-directpathfromonepointtoanothermayentailproceedingviacurvedRiemanniancoördinates,makingthat(perceived)pathlongerthanalternativepaths.Insum,thenewformalismpresentedhereisproposedasageneralmethodfordescribingandpredictingperceptualandcognitivesimilarityjudgments,asacomplementtostandardvectordistancemetrics(Euclidean,Mahalanobis,etc.),whichareapplicableonlytomeasuresinnon-curvedspaces.Theresultsareequallyapplicabletovisual,auditory,andothermodalities,aswellastoabstractconceptdata.Atthecoreoftheworkarethetwinprinciplesthati)sensorystimuli(andarbitrarydata)mayhaveinternalRiemannianstructure,i.e.,dependencerelationsamongtheir(dimensional)componentfeatures;andii)anysystem,naturalorartificial,thatprocessessuchdatacontainsintrinsicassumptionsorbiasesaboutthenatureofthosedependencerelations.SuchasystemmayassumethatinputdataareEuclideanandthattheircomponentsarethusindependent,orthesystemmayassumethepresenceofanyofaverywidevarietyofinter-componentdependencies(suchasneighborortopographyrelations).Weformalizesuchpremises,layinggroundworkforextendedstudyofnaturalperceptualsystemsandofartificialalgorithmsforprocessing,representing,andidentifyingstructureinarbitrarydata.Ongoingworkisfocusedonextendingthefindingstodomainsbeyondvision,withtheaimofidentifyingadditionalusefulapplicationsaswellasidentifyingfurtherfundamentalprinciplesofrepresentation.AcknowledgementsTheauthorsgratefullyacknowledgehelpfuldiscussionswithEliBowen.ThisresearchwassupportedinpartbygrantN00014-15-1-2132fromtheOfficeofNavalResearchandgrantN000140-15-1-2823fromtheDefenseAdvancedResearchProjectsAgency.

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page16of23

!a

!b

!c

d(!a,!b) :dE = 1.4;dH = 2;dR = 4.1

d(!a, !c) :dE = 1.4;dH = 2;dR = 4.6

d(!b, !c) :dE = 1.4;dH = 2;dR = 1.3

Figure1.IllustrationoftheRiemanniannatureofperceptualsimilarity.(Top)Thetransposesofthreevectors(100000),(000010),and(000001)(

!a,!b, !c )are

renderedasimageswithemptyspaceforzerosanddarkspotsforones.TheEuclidean

pairwisedistancesbetweenanytwoof !a , !b ,and

!c areequal(distancesof 2 ).TheirHammingdistancesalsoareequal(distancesof2).Ifwemeasurethedistancesbetweenthedarkspots,theanswers(inmm)comeouttobesimilarfrom

!a to !b andfrom

!a to !c ,

butquitedifferent(muchsmaller)from !b to

!c .This“rulerdistance”matchestheevokedperceptualsimilarityjudgmentsempiricallyelicitedfromhumanviewers:alljudge

!b and

!c tobemoresimilarthaneitheristo

!a .(Bottomleft)The64vectorsofthetwodimensionaldiscretecosinetransformformanorthogonalbasisinEuclideanspace;theyareequidistantfromeachother.Perceptualsimilarityjudgmentsbetweenthem,however,exhibitwidevariations;somearejudgedfarmoresimilartoeachotherthanothersbyhumanperceivers.(Bottomright)Takingjustthefirstand64thDCTentries(upperleftandlowerrightcornersoftheDCT,respectively)asanexample,whenviewedwithunitcoefficients(asontheleft),theyarejudgedquitedistinct;however,whenviewedwithintermediatecoefficientstheyarejudgedtobesomewhatsimilar(rightside).Thustheperceptualmetricbeingusedbyhumanviewersapparentlyisnotuniformacrossthisbasisspace.Thusnotonlyisthespacenon-Euclidean,italsoisnon-affine.Throughoutthispaper,weassumefullRiemanniancurvatureinthisbasisspace.

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page17of23

nn′n′′

n′n′′

n′′n′n

n′n′′

input output !x

!y

a) b)

d) n ′n ′′n 0 0 0 0 0′n n ′n ′′n 0 0 0 0′′n ′n n ′n ′′n 0 0 00 ′′n ′n n ′n ′′n 0 00 0 ′′n ′n n ′n ′′n 00 0 0 ′′n ′n n ′n ′′n0 0 0 0 ′′n ′n n ′n0 0 0 0 0 ′′n ′n n

⎡

⎣

⎢⎢⎢⎢⎢⎢⎢⎢⎢

⎤

⎦

⎥⎥⎥⎥⎥⎥⎥⎥⎥

e)

c)

receptive fields (Gaussian)

J(x)≡

∂y1∂x1

∂y1∂x2

!∂y1∂xp

∂y2∂x1

∂y2∂x2

!∂y2∂xp

" " ! "∂yq∂x1

∂yq∂x2

!∂yq∂xp

⎡

⎣

⎢⎢⎢⎢⎢⎢⎢⎢

⎤

⎦

⎥⎥⎥⎥⎥⎥⎥⎥

Figure2.BrainconnectomesareRiemannian.a)Simpleexampleofanatomicalprojectionsbetweentworegions.b)Simplevectorencodingofananatomicalprojectionwithsynapticweights.c)Examplesofphysiologicalneuralresponsesinearlyvisualareas(gaussians).d)AJacobianmatrixdenotingtheoveralleffectofactivityintheneuronsofaninputarea(x)ontheneuronsinatargetarea(f);eachentrydenotesthechangeinanelementoffasconsequenceofagivenchangeinanelementofx.e)ExampleinstanceofsuchaJacobian,correspondingtothesynapticconnectionpatterninpart(b).

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page18of23

physicalspace Φ

featurespace

µΦ→Ψ µF→Φ

perceptualspace (Z3)

Ψ(R3) (Z64)F

Figure3.Themapfromphysicaltoperceptualspace.Thethreerelevantprojectionspacesforimagecompression.Foralltheexamplesinthispaper,weadopttheJPEGassumptionofan8x8pixelimage.Theimageconsistsofasetofintensitysettingsforeachpixelatagivenxandycoordinate;thiscorrespondstoEuclidean“physicalspace”Φ .Imagesaremappedintofeaturespace,listingthe8x8pixelsasa64-dimensionalvectorwithintegerintensityvaluesfrom-255to+255.Humanjudgmentsofthesimilarityoftwoimages(suchasanoriginalandacompressedimage)correspondtoadistinct(Riemannian)spaceaccountingforgeometricneighborrelationsamongthepixels(absentfromfeaturespacerepresentation),alongwithjust-noticeabledifferences(JND)ofintensityvaluesatanygivenpixel.Themappingfunctions(µ )mapfromfeaturetophysicalspace( F →Φ )andfromphysicaltoperceptualspace(Φ→Ψ )asshown.

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page19of23

...

. . .. . .

a)

16 11 10 16 24 40 51 6112 12 14 19 26 58 60 5514 13 16 24 40 57 69 5614 17 22 29 51 87 80 6218 22 37 56 68 109 103 7724 35 55 64 81 104 113 9249 64 78 87 103 121 120 10172 92 95 98 112 100 103 99

⎡

⎣

⎢⎢⎢⎢⎢⎢⎢⎢⎢

⎤

⎦

⎥⎥⎥⎥⎥⎥⎥⎥⎥

QRGPEG =

15 35 35 40 40 42 42 4343 44 44 45 45 46 46 7890 90 93 93 97 97 99 99101 101 102 102 103 107 107 110111 111 114 114 115 115 116 116117 117 117 117 119 120 120 121121 122 122 125 125 125 126 126128 128 129 129 130 132 132 133

⎡

⎣

⎢⎢⎢⎢⎢⎢⎢⎢⎢

⎤

⎦

⎥⎥⎥⎥⎥⎥⎥⎥⎥

c)

g)

d) f)

e)

b)

Q JPEG

Q RGPEG

Figure4.Treatmentofimageasgraph,andderivationofHamiltonian.(a)Basisvectorsinfeaturespace F treatedasagraphwithwhosenodesarethedimensionsofthebasisandwhoseedgesarethepairwisedistancesbetweendimensions(seeEq(2.6)).Fromthatgraph,theadjacencyanddegreematrices,andthusthegraphLaplacian,canbedirectlycomputed.(b)QmatrixforJPEG(qualitylevel50%).(c)ComputedQmatrixforRGPEG.(d)HamiltonianforJPEG.(e)HamiltonianforRGPEG(seeSupplementalsection§2.7,table§5.(f)EigenvectorsofHamiltonianforJPEG.(g)EigenvectorsofHamiltonianforRGPEG.(SeeSupplementalsections§2.7-2.11).

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page20of23

Figure5.SidebysidecomparisonofJPEG(J)andRGPEG(R)compressiononasampleimage.(FormoreinstancesseeSupplementalfigures§15-§24.)(Top)Examplesofimages(alongsidecorrespondingcomputedJacobians)forgivenvaluesofdesiredquality(andcorrespondingQmatrices),atqualitylevels30,50,60,and80,forJPEG(J)andRGPEG(R).Foreachimage,thecomputederror(er)andentropy(en)aregivenbelowtheimage.Forcomparableerrormeasures,theentropyforRGPEGisconsistentlylowerthanforJPEG.(Bottomleft)Receiveroperatingcharacteristicforentropy-errortradeoffforJPEG(boxes)andRGPEG(circles).Atcomparableentropyvalues,RGPEGerrorvaluesareconsistentlyequivalentorsmaller.(Bottomright)Samplemeasuresofentropy(blue)anderror(purple)forJPEG(dotted)andRGPEG(solid)atdistinctqualitysettings.(AllimagesfromCaltech-256(37).

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page21of23

d3′

d1′d2′A

B

C

AB

C

d3

d1d2

perceptualmapping

stimulus input space

perceptual space

d3<d1+d2

d3′>d1′+d2′

Figure6.Interpretationofthetriangleinequalityviolation(initiallydescribedbyTverskyandGati1982).Inaphysicalstimulus,thedistancefromAtoBislessthanthecombineddistancesfromAtoCtoB,i.e., d3 ≤ d1 + d2 ,obeyingthetriangleinequalityinthestimulusinputspace.Aperceiver,however,measuresthosedistancesnotintheinputspacebutinherownperceptualreferenceframe,whichisaRiemannianspace(seetext).Thecurvatureofthatspacemayrenderdifferentgeodesicdistances;specifically,thegeodesicfromAtoBmaybelongerthanthegeodesicfromAtoCtoB;thus ′d3 > ′d1 + ′d2 ,violatingthetriangleinequalityinperceptualspace.

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page22of23

References1. TverskyA&GatiI(1982)Similarity,separability,andthetriangleinequality.

PsychologicalReview89:123-154.2. TverskyA(1977)Featuresofsimilarity.PsychologicalReview84:327-352.3. LoganJ,LivelyS,&PisoniD(1991)TrainingJapaneselistenerstoidentifyEnglish

/r/and/l/:afirstreport.JAcoustSocAm.89:874-886.4. KuhlP(2004)Earlylanguageacquisition:crackingthespeechcode.NatRev

Neurosci5:831-843.5. GuionS,FlegeJ,Akahane-YamadaR,&PruittJ(2000)Aninvestigationofcurrent

modelsofsecondlanguagespeechperception:thecaseofJapaneseadults'perceptionofEnglishconsonants..JAcoustSocAm.107:2711-2724.

6. IversonP,etal.(2003)Aperceptualinferenceaccountofacquisitiondifficultiesfornon-nativephonemes..Cognition87:B47-B57.

7. McCandlissB,FiezJ,ProtopapasA,ConwayM,&McClellandJ(2002)Successandfailureinteachingthe/r/-/l/contrasttoJapaneseadults:testsofaHebbianmodelofplasticityandstabilizationinspokenlanguageperception.CognAffectBehavNeurosci2:89-108.

8. NosofskyR(1992)Similarityscalingandcognitiveprocessmodels.AnnRevPsychol43:25-53.

9. NosofskyR&JohansenM(2000)Exemplar-basedaccountsof"multiplesystem"phenomenainperceptualcategorization.PsychonomicBulletin&Review7:375-402.

10. AshbyF&PerrinN(1988)Towardaunifiedtheoryofsimilarityandrecognition.PsychologicalReview95:124-150.

11. TodericiG,etal.(2016)Fullresolutionimagecompressionwithrecurrentneuralnetworks.arXiv.

12. SvobodaP,HradisM,BarinaD,&ZemcikP(2016)Compressionartifactsremovalusingconvolutionalneuralnetworks.arXiv.

13. WallaceG(1992)TheJPEGstillpicturecompressionstandard.IEEETrans.ConsumerElectronics38(1).

14. PennebakerW&MitchellJ(1993)JPEG:Stillimagedatacompressionstandard(VanNostrandReinhold).

15. KleinS,SilversteinD,&CarneyT(1992)Relevanceofhumanvisiontojpeg-dctcompression.SPIE/IS&TSymposiumonElectronicImaging10.1117/12.135968:200-215.

16. kishanA,LeeC,&WinerJ(2008)Branchedprojectionsintheauditorythalamocorticalandcorticocorticalsystems.Neuroscience154:283-293.

17. LeeC,SchreinerC,ImaizumiK,&WinerJ(2004)Tonotopicandheterotopicprojectionsystemsinphysiologicallydefinedauditorycortex.Neuroscience128:871-887.

18. Goldman-RakicP(1988)Topographyofcognition:paralleldistributednetworksinprimateassociationcortex.AnnRevNeurosci11:137-156.

19. WinkowskiD&KanoldP(2013)Laminartransformationoffrequencyorganizationinauditorycortex.JNeurosci33:1498-1508.

20. WatkinsP,KaoJ,&KanoldP(2014)Spatialpatternofintra-laminarconnectivityinsupragranularmouseauditorycortex.FrontiersinNeuralCircuits8:1-18.

21. EisenhartL(1997)Riemanniangeometry(PrincetonUniversityPress).22. OlshausenB&FieldD(1996)Emergenceofsimple-cellreceptivefieldpropertiesby

learningasparsecodefornaturalimages.Nature381:607-609.

Rodriguez&Granger Differentialgeometryofperceptualsimilarity[arXiv:1708.00138v1] BrainEngineeringLabTechReport2017.2

Page23of23

23. SoodakR(1986)Two-dimensionalmodelingofvisualreceptivefieldsusingGaussiansubunits.ProceedingsoftheNationalAcademyofScience83:9259-9263.

24. FieldD&TolhurstD(1986)Thestructureandsymmetryofsimple-cellreceptive-fieldprofilesinthecat'svisualcortex.ProcRoyalAcadSciB228:379-400.

25. RaoR&BallardD(1999)Predictivecodinginthevisualcortex:afunctionalinterpretationofsomeextra-classicalreceptive-fieldeffects.NatureNeurosci2(79-87).

26. LindebergA(2013)Acomputationaltheoryofvisualreceptivefields.BiologicalCybernetics107:589-635.

27. PeronaP&MalikJ(1990)Scale-spaceandedgedetecionusinganisotropicdiffusion.IEEETPAMI12:629-639.

28. MumfordD&ShahJ(1989)Optimalapproximationsbypiecewisesmoothfunctionsandassociatedvariationalproblems.CommunicationsonPureandAppliedMathematics42:577-685.

29. DudaR,HartP,&StorkD(2012)Patternclassification(Wiley).30. ShepardR(1962)Theanalysisofproximities:Multidimensionalscalingwithan

unknowndistancefunction(II).Psychometrika27:219-246.31. AttneaveF(1950)Dimensionsofsimilarity.AmericanJournalofPsychology63:516-

556.32. BrownE,NguyenD,FrankL,WilsonM,&SoloV(2001)Ananalysisofneural

receptivefieldplasticitybypointprocessadaptivefiltering.ProceedingsoftheNationalAcademyofScience98:12261-12266.

33. FritzJ,ShammaS,ElhilaliM,&KleinD(2003)Rapidtask-relatedplasticityofspectrotemporalreceptivefieldsinprimaryauditorycortex.NatureNeurosci6:1216-1223.

34. SmithE,PatalanoA,&JonidesJ(1998)Alternativestrategiesofcategorization.Cognition65:167-196.

35. EverittB(1993)ClusterAnalysis(EArnold).36. JainA(2010)Dataclustering:50yearsbeyondk-means.PatternRecogLett31:651-

666.37. GriffinG,HolubA,&PeronaP(TheCaltech256).(CaltechTechReport).