Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Geometric losses for distributional learning
Arthur Mensch(1), Mathieu Blondel(2), Gabriel Peyre(1)
(1) Ecole Normale Superieure, DMACentre National pour la Recherche Scientifique
Paris, France
(2) NTT Communication Science LaboratoriesKyoto, Japan
June 17, 2019
1 Introduction
2 Fenchel-Young losses for distribution spaces
3 Geometric softmax from Sinkhorn negentropy
4 Statistical properties
5 Applications
Introduction: Predicting distributions
Link
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 1 / 17
Introduction: Predicting distributions
Link
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 1 / 17
Introduction: Predicting distributions
Link
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 1 / 17
Introduction: Predicting distributions
Link
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 1 / 17
Introduction: Predicting distributions
Link
Loss
Link
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 1 / 17
Introduction: Predicting distributions
Link
Loss
Link
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 1 / 17
Contribution: losses and links for continuous metrized output
Handling output geometryLink and loss with cost between classes
C : Y × Y → ROutput distribution over continuous space Y
New geometric losses and associated link functions:1 Construction from duality between distributions and scores2 Need: Convex functional on distribution space
Provided by regularized optimal transport
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 2 / 17
Contribution: losses and links for continuous metrized output
Handling output geometryLink and loss with cost between classes
C : Y × Y → ROutput distribution over continuous space Y
New geometric losses and associated link functions:1 Construction from duality between distributions and scores
2 Need: Convex functional on distribution spaceProvided by regularized optimal transport
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 2 / 17
Contribution: losses and links for continuous metrized output
Handling output geometryLink and loss with cost between classes
C : Y × Y → ROutput distribution over continuous space Y
New geometric losses and associated link functions:1 Construction from duality between distributions and scores2 Need: Convex functional on distribution space
Provided by regularized optimal transport
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 2 / 17
Background: learning with a cost over outputs Y
Cost augmentation of losses1, 2:Convex cost-aware loss Lc : [1, d]× Rd → R
!4 Undefined link functions: Rd →4d: what to predict at test time ?
Use a Wasserstein distance between output distributions:Ground metric C defines a distance WC between distributionsPrediction with a softmax link
`(α, f) , WC(softmax(f), α))!4 Non-convex loss and costly to compute
1Ioannis Tsochantaridis et al. “Large margin methods for structured and interdependent output variables”. In: JMLR (2005).2Kevin Gimpel and Noah A Smith. “Softmax-margin CRFs: Training log-linear models with cost functions”. In: NAACL. 2010.
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 3 / 17
Background: learning with a cost over outputs Y
Cost augmentation of losses1, 2:Convex cost-aware loss Lc : [1, d]× Rd → R
!4 Undefined link functions: Rd →4d: what to predict at test time ?
Use a Wasserstein distance between output distributions3:Ground metric C defines a distance WC between distributionsPrediction with a softmax link
`(α, f) , WC(softmax(f), α))!4 Non-convex loss and costly to compute
1Ioannis Tsochantaridis et al. “Large margin methods for structured and interdependent output variables”. In: JMLR (2005).2Kevin Gimpel and Noah A Smith. “Softmax-margin CRFs: Training log-linear models with cost functions”. In: NAACL. 2010.3Charlie Frogner et al. “Learning with a Wasserstein loss”. In: NIPS. 2015.
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 3 / 17
1 Introduction
2 Fenchel-Young losses for distribution spaces
3 Geometric softmax from Sinkhorn negentropy
4 Statistical properties
5 Applications
Predicting distributions from topological duality
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 4 / 17
Predicting distributions from topological duality
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 4 / 17
Predicting distributions from topological duality
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 4 / 17
Predicting distributions from topological duality
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 4 / 17
Predicting distributions from topological duality
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 4 / 17
Predicting distributions from topological duality
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 4 / 17
All you need is a convex functional
Fenchel-Young losses4 5: Convex function Ω : 4d → R
and conjugateΩ?(f) = min
α∈4dΩ(α)− 〈α, f〉 `Ω(α, f) = Ω(α) + Ω?(f)− 〈α, f〉 > 0
Define link functions between dual and primal
4John C. Duchi et al. “Multiclass Classification, Information, Divergence, and Surrogate Risk”. In: Annals of Statistics (2018).5Mathieu Blondel et al. “Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithms”. In: AISTATS. 2019.
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 5 / 17
All you need is a convex functional
Fenchel-Young losses4 5: Convex function Ω : 4d → R and conjugateΩ?(f) = min
α∈4dΩ(α)− 〈α, f〉
`Ω(α, f) = Ω(α) + Ω?(f)− 〈α, f〉 > 0
Define link functions between dual and primal
4John C. Duchi et al. “Multiclass Classification, Information, Divergence, and Surrogate Risk”. In: Annals of Statistics (2018).5Mathieu Blondel et al. “Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithms”. In: AISTATS. 2019.
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 5 / 17
All you need is a convex functional
Fenchel-Young losses4 5: Convex function Ω : 4d → R and conjugateΩ?(f) = min
α∈4dΩ(α)− 〈α, f〉 `Ω(α, f) = Ω(α) + Ω?(f)− 〈α, f〉 > 0
Define link functions between dual and primal
4John C. Duchi et al. “Multiclass Classification, Information, Divergence, and Surrogate Risk”. In: Annals of Statistics (2018).5Mathieu Blondel et al. “Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithms”. In: AISTATS. 2019.
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 5 / 17
All you need is a convex functional
Fenchel-Young losses4 5: Convex function Ω : 4d → R and conjugateΩ?(f) = min
α∈4dΩ(α)− 〈α, f〉 `Ω(α, f) = Ω(α) + Ω?(f)− 〈α, f〉 > 0
Define link functions between dual and primal∇Ω(α) = argmin
f∈Rd
`Ω(α, f) ∇Ω?(f) = argminα∈4d
`Ω(α, f)4John C. Duchi et al. “Multiclass Classification, Information, Divergence, and Surrogate Risk”. In: Annals of Statistics (2018).5Mathieu Blondel et al. “Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithms”. In: AISTATS. 2019.
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 5 / 17
All you need is a convex functional
Fenchel-Young losses4 5: Convex function Ω : 4d → R and conjugateΩ?(f) = min
α∈4dΩ(α)− 〈α, f〉 `Ω(α, f) = Ω(α) + Ω?(f)− 〈α, f〉 > 0
Define link functions between dual and primal`Ω(α, f) = 0⇔ α = ∇Ω∗(f) ∇Ω?(f) = argmin
α∈4d
`Ω(α, f)4John C. Duchi et al. “Multiclass Classification, Information, Divergence, and Surrogate Risk”. In: Annals of Statistics (2018).5Mathieu Blondel et al. “Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithms”. In: AISTATS. 2019.
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 5 / 17
Discrete canonical example: Shannon entropy
Ω(α) = −H(α) =d∑i=1
αi logαi Ω∗(f) = logsumexp(f)
Not defined on continuous distributions, cost-agnosticWe need an entropy defined on all distributions
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 6 / 17
Discrete canonical example: Shannon entropy
Ω(α) = −H(α) =d∑i=1
αi logαi Ω∗(f) = logsumexp(f)
Not defined on continuous distributions, cost-agnosticWe need an entropy defined on all distributions
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 6 / 17
Discrete canonical example: Shannon entropy
Ω(α) = −H(α) =d∑i=1
αi logαi Ω∗(f) = logsumexp(f)
Not defined on continuous distributions, cost-agnostic
We need an entropy defined on all distributions
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 6 / 17
Discrete canonical example: Shannon entropy
Ω(α) = −H(α) =d∑i=1
αi logαi Ω∗(f) = logsumexp(f)
Not defined on continuous distributions, cost-agnosticWe need an entropy defined on all distributions
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 6 / 17
1 Introduction
2 Fenchel-Young losses for distribution spaces
3 Geometric softmax from Sinkhorn negentropy
4 Statistical properties
5 Applications
Detour: regularized optimal transport6
Distance between distribution on YOTC,ε(α, β) = min
π∈U(α,β)〈C, π〉+εKL(π|α⊗β)
Displacement cost for optimal couplingU(α, β)=π ∈M+
1 (Y2), π1 = α, π>1 = β
+ Entropic regularization: spread coupling
6Marco Cuturi. “Sinkhorn distances: Lightspeed computation of optimal transport”. In: NIPS. 2013.
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 7 / 17
Sinkhorn negentropy from regularized optimal transport
Self regularized optimal transportation distance:
ΩC(α) = −12OTC,ε=2(α, α) = − max
f∈C(Y)〈α, f〉 − log〈α⊗ α, exp(f ⊕ f − C2 )〉
Continuous convex
Special cases
εΩC/ε(α) ε→+∞−→ 12〈α⊗ α,−C〉 MMD
C =
0 ∞ . . .
∞ 0 . . .
Shannon entropyGini index
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 8 / 17
Sinkhorn negentropy from regularized optimal transport
Self regularized optimal transportation distance:
ΩC(α) = −12OTC,ε=2(α, α) = − max
f∈C(Y)〈α, f〉 − log〈α⊗ α, exp(f ⊕ f − C2 )〉
Continuous convex
Special cases
εΩC/ε(α) ε→+∞−→ 12〈α⊗ α,−C〉 MMD
C =
0 ∞ . . .
∞ 0 . . .
Shannon entropyGini index
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 8 / 17
Sinkhorn negentropy from regularized optimal transport
Self regularized optimal transportation distance:
ΩC(α) = −12OTC,ε=2(α, α) = − max
f∈C(Y)〈α, f〉 − log〈α⊗ α, exp(f ⊕ f − C2 )〉
Continuous convex
Special cases
εΩC/ε(α) ε→+∞−→ 12〈α⊗ α,−C〉 MMD
C =
0 ∞ . . .
∞ 0 . . .
Shannon entropyGini index
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 8 / 17
Dual mapping from Sinkhorn negentropy
Ω(α) = − maxf∈C(Y)
〈α, f〉 − log〈α⊗ α, exp(f ⊕ f − C2 )〉
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 9 / 17
Dual mapping from Sinkhorn negentropy
∇Ω(α) = − argmaxf∈C(Y)
〈α, f〉 − log〈α⊗ α, exp(f ⊕ f − C2 )〉
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 9 / 17
Returning to primal: geometric softmax
Ω∗ = g-logsumexp : f 7→ − log minα∈M+
1 (Y)〈α⊗ α, exp(−f ⊕ f + C
2 )〉
Minimizes a simple quadratic.
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 10 / 17
Returning to primal: geometric softmax
∇Ω∗ = g-softmax : f 7→ argminα∈M+
1 (Y)〈α⊗ α, exp(−f ⊕ f + C
2 )〉
Minimizes a simple quadratic.
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 10 / 17
Geomtric loss construction and computation
Training with the geometric logistic loss:
`C(α, f) = geometric-LSEC(f) + sinkhorn-negentropyC(α)− 〈α, f〉
Tractable discrete Y: use mirror descent/L-BFGS10× as costly as a softmaxBackpropagation: ∇Ω? = geometric-softmax
Continuous case:Frank-Wolfe scheme, adding one Dirac at each iteration
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 11 / 17
Properties of the geometric-softmax
−2 0 2
f1 − f2
0.0
0.2
0.4
0.6
0.8
1.0
∇Ω?(f
) 1C =(0 γ
γ 0
)
0.00 0.25 0.50 0.75 1.00
α
0.0
0.2
0.4
0.6
−Ω
([α,1−α
])
γ =∞γ = 0.1
γ = 2
∇Ω? returns from Sinkhorn potentials: ∇Ω? ∇Ω = Id
∇Ω? non injective (∼ softmax)∇Ω ∇Ω∗ projects f onto symmetric Sinkhorn potentialsSparse ∇Ω∗(f) ← minimization on the simplex
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 12 / 17
Properties of the geometric-softmax
−2 0 2
f1 − f2
0.0
0.2
0.4
0.6
0.8
1.0
∇Ω?(f
) 1C =(0 γ
γ 0
)
0.00 0.25 0.50 0.75 1.00
α
0.0
0.2
0.4
0.6
−Ω
([α,1−α
])
γ =∞γ = 0.1
γ = 2
∇Ω? returns from Sinkhorn potentials: ∇Ω? ∇Ω = Id∇Ω? non injective (∼ softmax)∇Ω ∇Ω∗ projects f onto symmetric Sinkhorn potentialsSparse ∇Ω∗(f) ← minimization on the simplex
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 12 / 17
Properties of the geometric-softmax
ε∇Ω∗(f/ε)ε→ 0 Mode finding ε→∞ Positive deconvolution
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 13 / 17
1 Introduction
2 Fenchel-Young losses for distribution spaces
3 Geometric softmax from Sinkhorn negentropy
4 Statistical properties
5 Applications
Consistent learning with the geometric logistic loss
Bregman divergence from Sinkhorn negentropy:Dg(α|β) = Ω(α)− Ω(β)− 〈∇Ω(α), α− β〉.
= asymmetric transport distance
Sample distribution (xi, αi)i ∈ X ×M+1 (Y).
Fisher consistency:min
β:X→M+1 (Y)
E [Dg (α, β (x))] = ming:X→C(Y)
E [`g (α, g-softmax (g (x)))]
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 14 / 17
Consistent learning with the geometric logistic loss
Bregman divergence from Sinkhorn negentropy:Dg(α|β) = Ω(α)− Ω(β)− 〈∇Ω(α), α− β〉.
= asymmetric transport distance
Sample distribution (xi, αi)i ∈ X ×M+1 (Y).
Fisher consistency:min
β:X→M+1 (Y)
E [Dg (α, β (x))] = ming:X→C(Y)
E [`g (α, g-softmax (g (x)))]
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 14 / 17
Consistent learning with the geometric logistic loss
Bregman divergence from Sinkhorn negentropy:Dg(α|β) = Ω(α)− Ω(β)− 〈∇Ω(α), α− β〉.
= asymmetric transport distance
Sample distribution (xi, αi)i ∈ X ×M+1 (Y).
Fisher consistency:min
β:X→M+1 (Y)
E [Dg (α, β (x))] = ming:X→C(Y)
E [`g (α, g-softmax (g (x)))]
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 14 / 17
1 Introduction
2 Fenchel-Young losses for distribution spaces
3 Geometric softmax from Sinkhorn negentropy
4 Statistical properties
5 Applications
Applications: variational auto-encoder
Goal: Generate nearly 1D distribution in 2D imagesDataset: Google QuickdrawTraditional sigmoid activation layer
← replaced by geometric softmax
Deconvolutional effectCost-informed non-linearity
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 15 / 17
Applications: variational auto-encoder
Goal: Generate nearly 1D distribution in 2D imagesDataset: Google QuickdrawTraditional sigmoid activation layer
← replaced by geometric softmax
Deconvolutional effectCost-informed non-linearity
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 15 / 17
Applications: variational auto-encoders
softmax g-softmax
Reconstruction
Generation
Better defined generated imagesArthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 16 / 17
Conclusion
Geometric softmax:New loss and projector onto output probabilitiesDiscrete/continuous, aware of a cost between outputsFenchel duality in Banach spaces + regularized optimal transportApplication in VAE and ordinal regressiongithub.com/arthurmensch/g-softmax
Future directions:How to improve computation methods (continuous FW)Geometric logisitic loss in super resolution
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 17 / 17
Conclusion
Geometric softmax:New loss and projector onto output probabilitiesDiscrete/continuous, aware of a cost between outputsFenchel duality in Banach spaces + regularized optimal transportApplication in VAE and ordinal regressiongithub.com/arthurmensch/g-softmax
Future directions:How to improve computation methods (continuous FW)Geometric logisitic loss in super resolution7
7Nicholas Boyd et al. “DeepLoco: Fast 3D localization microscopy using neural networks”. In: BioRxiv (2018), p. 267096.
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 17 / 17
Mathieu Blondel Gabriel Peyre
Poster # 179
Arthur Mensch, Mathieu Blondel, Gabriel Peyre Geometric losses for distributional learning 17 / 17