Metric learning for diffeomorphic image registration. · Vialard Introduction to di eomorphisms group and Riemannian tools Choice of the metric Spatially dependent metrics Metric

Metric learning fordiffeomorphic image

registration.

François-XavierVialard

Metric learning for diffeomorphic imageregistration.

François-Xavier Vialard

Université Paris-Est Marne-la-Vallée

joint work with M. Niethammer and R. Kwitt.

IHP, March 2019.


registration.


Outline

1 Introduction to diffeomorphisms group and Riemannian tools

2 Choice of the metric

3 Spatially dependent metrics

4 Metric learning

5 SVF metric learning


registration.


Introduction todiffeomorphisms groupand Riemannian tools

Choice of the metric

Spatially dependentmetrics

Metric learning

SVF metric learning

Example of problems of interest

Given two shapes, find a diffeomorphism of R3 that maps oneshape onto the other


registration.





Metric learning

SVF metric learning

Example of problems of interest

Given two shapes, find a diffeomorphism of R3 that maps oneshape onto the other

Different data types and different way of representing them.

Figure – Two slices of 3D brain images of the same subject at differentages


registration.





Metric learning

SVF metric learning

Example of problems of interestGiven two shapes, find a diffeomorphism of R3 that maps oneshape onto the other

Deformation by a diffeomorphism

Figure – Diffeomorphic deformation of the image

SimulationBrain.mp4Media File (video/mp4)


registration.





Metric learning

SVF metric learning

Variety of shapes

Figure – Different anatomical structures extracted from MRI data


registration.





Metric learning

SVF metric learning

A Riemannian approach to diffeomorphicregistration

Several diffeomorphic registration methods are available:

• Free-form deformations B-spline-based diffeomorphisms by D.Rueckert

• Log-demons (X.Pennec et al.)• Large Deformations by Diffeomorphisms (M. Miller,A.

Trouvé, L. Younes)

• ANTSOnly the two last ones provide a Riemannian framework.


registration.





Metric learning

SVF metric learning

A Riemannian approach to diffeomorphicregistration

• vt ∈ V a time dependent vector field on Rn.• ϕt ∈ Diff , the flow defined by

∂tϕt = vt(ϕt) . (1)

Action of the group of diffeomorphism G0 (flow at time 1):

Π : G0 × C → C ,Π(ϕ,X )

.= ϕ.X

Right-invariant metric on G0: d(ϕ0,1, Id)2 = 12

∫ 10|vt |2V dt.

−→ Strong metric needed on V(Mumford and Michor: Vanishing Geodesic Distance on...)


registration.





Metric learning

SVF metric learning

Matching problems in a diffeomorphic framework

1 U a domain in Rn

2 V a Hilbert space of C 1 vector fields such that:

‖v‖1,∞ 6 C |v |V .

V is a Reproducing kernel Hilbert Space (RKHS): (pointwiseevaluation continuous)=⇒ Existence of a matrix function kV (kernel) defined on U × Usuch that:

〈v(x), a〉 = 〈kV (., x)a, v〉V .

Right invariant distance on G0

d(Id, ϕ)2 = infv∈L2([0,1],V )

∫ 10

|vt |2V dt ,

−→ geodesics on G0.


registration.





Metric learning

SVF metric learning

Variational formulation

Find the best deformation, minimize

J (ϕ) = infϕ∈GV

d(ϕ.A,B)2︸︷︷︸similarity measure

(2)


registration.





Metric learning

SVF metric learning

Variational formulation

Find the best deformation, minimize

��

��J (ϕ) = inf

ϕ∈GVd(ϕ.A,B)2︸︷︷︸

similarity measure

(2)

Tychonov regularization:

J (ϕ) = R(ϕ)︸︷︷︸Regularization

+1

2σ2d(ϕ.A,B)2︸︷︷︸

similarity measure

. (3)

Riemannian metric on GV :

R(ϕ) =1

2

∫ 10

|vt |2V dt (4)

is a right-invariant metric on GV .


registration.





Metric learning

SVF metric learning

Optimization problem

Minimizing

J (v) = 12

∫ 10

|vt |2V dt +1

2σ2d(ϕ0,1.A,B)

2 .

In the case of landmarks:

J (ϕ) = 12

∫ 10

|vt |2V dt +1

2σ2

k∑i=1

‖ϕ(xi )− yi‖2 ,

In the case of images:

d(ϕ0,1.I0, Itarget)2 =

∫U

|I0 ◦ ϕ1,0 − Itarget |2dx .

Main issues for practical applications:

• choice of the metric (prior),• choice of the similarity measure.


registration.





Metric learning

SVF metric learning

Why does the Riemannian framework matter?

Generalizations of statistical tools in Euclidean space:

• Distance often given by a Riemannian metric.• Straight lines → geodesic defined by

Variational definition: arg minc(t)

∫ 10

‖ċ‖2c(t) dt = 0 ,

Equivalent (local) definition: ∇ċ ċ = c̈ + Γ(c)(ċ , ċ) = 0 .

• Average → Fréchet/Karcher mean.

Variational definition: arg min{x → E [d2(x , y)]dµ(y)}Critical point definition: E [∇x d2(x , y)]dµ(y)] = 0 .

• PCA → Tangent PCA or PGA.• Geodesic regression, cubic regression...(variational or

algebraic)


registration.





Metric learning

SVF metric learning

Karcher mean on 3D images

Init. guesses

1 iteration

2 iterations

3 iterationsA1i A

2i A

3i A

4i

Figure – Average image estimates Ami , m ∈ {1, · · · , 4} after i =0, 1, 2and 3 iterations.


registration.





Metric learning

SVF metric learning

Interpolation, Extrapolation

Figure – Geodesic regression (MICCAI 2011)


registration.





Metric learning

SVF metric learning

Interpolation, Extrapolation

Figure – Extrapolation of happiness


registration.





Metric learning

SVF metric learningWhat metric to choose?


registration.





Metric learning

SVF metric learning

Choosing the right-invariant metricRight-invariant metric: Eulerian fluid dynamic viewpoint onregularization.Space V of vector fields is defined equivalently by

• its kernel K such as Gaussian kernel,• its differential operator, for instance (Id−σ∆)n for Sobolev

spaces.

The norm on V is simply

‖v‖2V =∫

Ω

〈v(x), (Lv)(x)〉 dx =∫

Ω

(L1/2v)2(x) dx .

Scale parameter important!

kσ(x , y) = e− ‖x−y‖

2

σ2 kernel/operator (Id−σ∆)n (5)

• σ small: good matching but non regular deformations andmore local minima.

• σ large: poor matching but regular deformations and moreglobal minima.


registration.





Metric learning

SVF metric learning

Sum of kernels and multiscaleChoice of mixture of Gaussian kernels: (Risser, Vialard et al. 2011)

K (x , y) =n∑

i=1

αi e− ‖x−y‖

2

σ2i (6)

Figure – Left to right: Small scale, large scale and multi scale.


registration.





Metric learning

SVF metric learning

Decomposition over different scales

Try to disentangle contributions at each scale: (Bruveris, Risser,Vialard, 2012, Siam MMS) using semi-direct product of groups.Consider Gσ1 ,Gσ2 two diffeomorphism groups at different scalesassociated with Vσ1 and Vσ2 .

Semi-direct product of groups Gσ1 n Gσ2 .

Non-linear extension of the infimal convolution of norms:

‖v‖2 = min(v1,v2)∈V1×V2

{‖v1‖2V1 + ‖v2‖

2V2

∣∣∣ v = v1 + v2} . (7)Non-linear extension −→ semi-direct product of groups.


registration.





Metric learning

SVF metric learning

From Eulerian to Lagrangian viewpoints

Spatial correlation of the deformation: need for local deformabilityon the tissues.Toward a more Lagrangian point of view.

How to introduce spatially varying metric?

Using kernels: χi being a partition of unity of the domain.

K =n∑

i=1

χi Kiχi , . (8)

This kernel is associated to the following variational interpretation:

‖v‖2 = min(v1,...,vn)∈V1×...×Vn

{n∑

i=1

‖vi‖2Vi∣∣∣ n∑

i=1

χi vi = v

}. (9)

−→ possibility to introduce soft-symmetries...


registration.





Metric learning

SVF metric learning

Left-invariant metrics

Miccai 2013, Marsden’s Fields volume, Schmah, Risser, VialardChange of point of view: choose body-coordinates and convectivevelocity:

J (ϕ) = 12

∫ 10

‖v(t)‖2V dt + E (ϕ(1) · I , J), (10)

under the convective velocity constraint:

∂tϕ(t) = dϕ(t) · v(t) , (11)

where dϕ(t) is the tangent map of ϕ(t).

• More natural interpretation of spatially varying metrics.• Left action + left invariant metric =⇒ no induced

Riemannian metric.


registration.





Metric learning

SVF metric learning

Difference with LDDMMThe path look different:

Figure – The green curves Right-LDM geodesic path, The blue curvesLeft-LDM geodesic path.

LDDMM LIDM


registration.





Metric learning

SVF metric learning

What’s next

Left-invariance is more Lagrangian but the metric is fixed as in theEulerian situation!

• On a template, learning the metric.

Motivation:

• Better matching results: i.e better regularization or matching.• Better matching quality for organs with (segmented) tumors.


registration.





Metric learning

SVF metric learning

Metric learning

Figure – Given a collection of shape and a template, learn the metric.


registration.





Metric learning

SVF metric learning

Metric learning: High-dimensional inverseproblem

(Miccai 2014: Vialard, Risser)

• (In)n=1,...,N be a population of N images.• T be a template (Karcher mean for instance).

Registering the template T onto the image In consists inminimizing:

JIn,K (v) =1

2

∫ 10

‖v(t)‖2V dt + d(T ◦ ϕ(1)−1, In) , (12)

where∂tϕ(t) = v(t) ◦ ϕ(t) .

Equivalent to minimize

JIn,K (v) =1

2

∫ 10

〈P(t)∇I (t),K?(P(t)∇I (t)〉 dt+d(T◦ϕ(1)−1, In) ,

(13)


registration.





Metric learning

SVF metric learning

Optimize over K? Ill posed!• Incorporate the smoothness constraint by defining

K = {K̂ MK̂ |M SDP operator on L2(Rd ,Rd )} , (14)

M symmetric positive definite matrix.

• Regularization on M. Prior for M close to Id. Minimize

F(M) = β2

d2S++ (M, Id) +1

N

N∑n=1

minvJIn (v ,M) , (15)

where d2 can be chosen as

• Affine invariant metric (Pennec et al.)g1 = Tr(M

−1(δM)M−1(δM)).

• (Modified) Wasserstein metric.

Problem

Problem: matrix M is huge: (dn)2 where d = 2, 3 dimension andn number of voxels.Computing the logarithm is costly.


registration.





Metric learning

SVF metric learning

Optimize over K? Ill posed!• Incorporate the smoothness constraint by defining

K = {K̂ MK̂ |M SDP operator on L2(Rd ,Rd )} , (14)

M symmetric positive definite matrix.• Regularization on M. Prior for M close to Id. Minimize

F(M) = β2

d2S++ (M, Id) +1

N

N∑n=1

minvJIn (v ,M) , (15)

where d2 can be chosen as

• Affine invariant metric (Pennec et al.)g1 = Tr(M

−1(δM)M−1(δM)).

• (Modified) Wasserstein metric.

Problem

Problem: matrix M is huge: (dn)2 where d = 2, 3 dimension andn number of voxels.Computing the logarithm is costly.


registration.





Metric learning

SVF metric learning

Wasserstein metric

Pros: Easy to computeCons: Non complete metric.

Trick

The map N 7→ NNT is a Riemannian submersion from Mn(R)equipped with the Frobenius norm to the space of SDP matricesequipped with the Wasserstein metric

Encode the symmetric matrix as NNT and perform theoptimization on N. The regularization reads 12‖N − Id‖

2

The gradient is

∇L2F(N) = β(N − Id)− (16)

1

2N

N∑n=1

∫ 10

(K̂ ? Pn(t))⊗ (NK̂ ? Pn(t)) + (NK̂ ? Pn(t))⊗ (K̂ ? Pn(t))dt ,

(17)


registration.





Metric learning

SVF metric learning

Reducing the problem dimension

IdeaLearn at large scales and not at fine scale.

Introduce:

K = {K̂ MΠK̂ + K̂ (Id − Π)K̂ |M SDP operator on L2(Rd ,Rd )} .(18)

with Π an orthogonal projection on a finite dimensionalparametrization of vector fields: use of splines.


registration.





Metric learning

SVF metric learning

Experiments• 40 subjects of the LONI Probabilistic Brain Atlas (LPBA40).• All 3D images were affinely aligned to subject 5 using ANTS.

Table – Reference results

No Reg SyN Kfine Kref

TO 0.665 0.750 0.732 0.712

DetJMax 1 3.17 4.65 1.66DetJMin 1 0.047 0.46 0.67DetJStd 0 0.17 0.11 0.063

Table – Average results.

DiagM GridM1 GridM2 K20 K30

TO 0.711 0.710 0.704 0.710 0.704

DetJMax 1.66 1.61 1.41 1.62 1.50DetJMin 0.68 0.70 0.67 0.73 0.66DetJStd 0.062 0.059 0.049 0.056 0.063

For a given quality of overlap, better smoothness of thedeformations.


registration.





Metric learning

SVF metric learning

Main issues

• The metric is fixed in Eulerian coordinates.• The metric is template based.

Make the method adaptive to any pairs of images on a simplermodel.


registration.





Metric learning

SVF metric learning

SVF model: A simple model

Based on Metric learning for image registration, CVPR 2019,Niethammer, Kwitt, Vialard.Let v(x) be a vector field. Find a v minimizer of

1

2‖v‖2V + Sim(I ◦ ϕ−11 , J) (19)

∂tϕt = v(ϕt) . (20)

Equivalent momentum formulation: Find m ∈ L2(Ω,Rd ) ⊂ V ∗such that

1

2〈m,K ?m〉+ Sim(I ◦ ϕ−11 , J) (21)

s.t.∂tϕ−1(t, x) + Dϕ−1(t, x)(K ?m(t, x)) = 0 . (22)

Numerical discretization: Central differences in space and 20timesteps in time of RK4.


registration.





Metric learning

SVF metric learning

Parametrization of the metricFix a collection of scales σ0 < . . . < σN−1 and set

Gi (x) = e−|x|2/σ2i .

v0(x)def.= (K (w) ?m0)(x)

def.=

N−1∑i=0

√wi (x)

∫y

Gi (|x − y |)√

wi (y)m0(y) dy , (23)

Problem: wi should be sufficiently smooth to guaranteediffeomorphisms.Introduce pre-weights ωi (x) and fix Kσ, with σ small:

Kσ ? ωi = wi

and ∑i

wi (x) = 1 .

Learning the metric is still ill-posed:

ÔMT(w) =

∣∣∣∣log σN−1σ0∣∣∣∣−r N−1∑

i=0

wi

∣∣∣∣log σN−1σi∣∣∣∣r (24)

Is 0 for (wi ) = (0, . . . , 0, 1).


registration.





Metric learning

SVF metric learning

Objective function

Optimize momentum + metric.

Obj0,1(m, ω) = argminm0

λ〈m0, v0〉 + Sim[I0 ◦ Φ−1(1), I1] +

λOMT

∫ÔMT(w(x)) dx +

λTV

√√√√N−1∑l=0

(∫γ(‖∇I0(x)‖)‖∇ωl (x)‖2 dx

)2, (25)

where γ(x) ∈ R+ is an edge indicator function

γ(‖∇I‖) = (1 + α‖∇I‖)−1, α > 0 .

Then, minimize ∑i,j

Obj0,1(mi,j , ωi ) , (26)

where ωi = fθ(Ii ).


registration.





Metric learning

SVF metric learning

Parametrize and learn the pre-weights ωi

The pre-weights are parametrized by a 2-layers net:

(ωi )i=1,...,N = ShallowNet(I ) .

Input: current image, Output: pre-weights.

ShallowNet = conv(d , n1) → BatchNorm → lReLU →conv(n1,N) → BatchNorm → weighted-linear-softmax

σw (z)j =clamp0,1(wj + zj − z)∑N−1

i=0 clamp0,1(wi + zi − z), (27)

The weights wj are reasonably initialized: wi = σ2i /(∑N−1

j=0 σ2j )


registration.





Metric learning

SVF metric learning

Optimization

Shared parameters: ShallowNetwork parameters,Individual parameters: Momentum for each pair.

1 (1) initialize with reasonable weights and optimize overmomentums,

2 (2) Jointly optimize on the shared and individual parameters:use SGD with (Nesterov) momentum, different batch size in2d/3d, 50 epochs in 2d, less in 3D.


registration.





Metric learning

SVF metric learning

Experiments on synthetic data1) Generate concentric circular regions with random radii and

associate different multi-Gaussian weights to these regions.We associate a fixed multi-Gaussian weight to thebackground.

2) Randomly create vector momenta at the borders of theconcentric circles.

3) Add noise (for texture) and compute forward model, toobtain source image, similar for target image.

0

1

2

3

disp

erro

r (es

t-GT)

[pix

el]

glob

alt=

0.01

;o=5

_l

t=0.

01;o

=15_

l

t=0.

01;o

=25_

l

t=0.

01;o

=50_

l

t=0.

01;o

=75_

l

t=0.

01;o

=100

_l

t=0.

10;o

=5_l

t=0.

10;o

=15_

l

t=0.

10;o

=25_

l

t=0.

10;o

=50_

l

t=0.

10;o

=75_

l

t=0.

10;o

=100

_l

t=0.

25;o

=5_l

t=0.

25;o

=15_

l

t=0.

25;o

=25_

l

t=0.

25;o

=50_

l

t=0.

25;o

=75_

l

t=0.

25;o

=100

_l0.0

0.5

1.0

1.5

2.0

disp

erro

r (es

t-GT)

[pix

el]

Figure – Displacement error (in pixel) with respect to the ground truthfor various values of the total variation penalty, λTV (t) and the OMTpenalty, λOMT (o). Results for the interior (top) and the exterior(bottom) rings show subpixel registration accuracy for all local metricoptimization results (* l). Overall, local metric optimizationsubstantially improves registrations over the results obtained via initialglobal multi-Gaussian regularization (global).


registration.





Metric learning

SVF metric learning

Experiments on synthetic dataSource image Target image

Warped source Deformation grid Std.dev.

λOMT = 150.165

0.170

0.175

0.180

0.185

0.190

0.195

λOMT = 500.170

0.175

0.180

0.185

0.190

0.195

λOMT = 100 0.1650.170

0.175

0.180

0.185

0.190

0.195

Figure – λTV = 0.1. Overall variance is similar but the true weights arenot recovered: weights on the outer ring [0.05, 0.55, 0.3, 0.1]


registration.





Metric learning

SVF metric learning

Experimentsw0(σ = 0.01) w1(σ = 0.05) w2(σ = 0.10) w3(σ = 0.20)

λOMT = 15

λOMT = 50

λOMT = 100

Figure


registration.





Metric learning

SVF metric learning

On 2D real data: LPBA40

Source image Target image

Warped source Deformation grid Std. dev.

λOMT = 15 0.1910.192

0.193

0.194

0.195

0.196

0.197

λOMT = 50 0.1920.193

0.194

0.195

0.196

0.197

0.198

λOMT = 1000.194

0.195

0.196

0.197

0.198

0.199


registration.





Metric learning

SVF metric learning

Performance on 3D data: CUMC12

Training on different dataset:

Trained on different dataset, test on CUMC12

Training 132 image pairs on CUMC12, 90 image pairs onMGH10, 150 image pairs on IBSR18.

Method mean std 1% 5% 50% 95% 99% p MW-stat sig?FLIRT 0.394 0.031 0.334 0.345 0.396 0.442 0.463


registration.





Metric learning

SVF metric learning

Conclusion

Summary

Adaptive metric learning in SVF.

Avoid end to end training for preserving diffeomorphicproperties.

Diffeomorphic guarantees at test time (no guarantee for DLmethods: VoxelMorph).

Perspectives

Combine it with momentum prediction (QuickSilver like).

Use it in LDDMM.

Incorporate richer deformations descriptors.

Paper to appear: Metric learning for image registration, CVPR2019, Niethammer, Kwitt, Vialard.

Main TalkIntroduction to diffeomorphisms group and Riemannian toolsChoice of the metricSpatially dependent metricsMetric learningSVF metric learning

anm0: anm1:

Documents

Metric learning for diffeomorphic image registration. · Vialard Introduction to di eomorphisms group and Riemannian tools Choice of the metric Spatially dependent metrics Metric