Spherical CNNs · 2020-01-26 · Rotated MNIST on the sphere Recognition of 3D shapes Prediction of...

Spherical CNNs

Credit: Taco S. Cohen1, Mario Geiger2, Jonas Kohler1, Max Welling1,3

1University of Amsterdam

3CIFAR

Presenter: Fuwen Tanhttps://qdata.github.io/deep2Read

Credit: Taco S. Cohen, Mario Geiger, Jonas Kohler, Max Welling (shortinst)Spherical CNNs Presenter: Fuwen Tan https://qdata.github.io/deep2Read 1 / 32

Outline

1 Background reading

2 GoalsCNNs on planar images → CNNs on spherical images

3 Equivariance propertiesPlanar CNN is translation-equivariantSpherical CNN is rotation-equivariant

4 Implementation

5 ExperimentsEquivariance errorRotated MNIST on the sphereRecognition of 3D shapesPrediction of atomization energies from molecular geometry

6 Take-home messages

If you get confused

Group Equivariant Convolutional Networks [1]T.S. Cohen, M. WellingICML, 2016.

Outline

4 Implementation

Parameterization

Plane : x(u, v) ∈ R2

Sphere : x(α, β) = Z (α)Y (β)n ∈ S2

n : north pole

3D Rotation : R(α, β, γ) = Z (α)Y (β)Z (γ) ∈ SO(3)

Figure: Proper Euler angles geometrical definition. The xyz (fixed) system isshown in blue, the XYZ (rotated) system is shown in red. The line of nodes (N)is shown in green. Credit: https://en.wikipedia.org/wiki/Euler_angles

CNNs on planar images

(f ∗ ψ)(x) =

f (y)ψ(x − y)dy

f : R2 → R (e.g . feature Maps)

ψ : R2 → R (e.g . locally − supported filter)

CNNs on planar images

(f ∗ ψ)(x) =

f (y)ψ(T−1x (y))dy

Tx(t) = t + x (translation)

T−1x (t) = x − t

CNNs on spherical images (first layer)

(f ∗ ψ)(x) =

f (y)ψ(R−1x (y))dy

x , y : 3D unit vector ∈ S2

Rx(t) = Rx · t (3Drotation)

Rx : (α, β, γ) ∈ SO(3)

A subtitle point

(f ∗ ψ)(x) =

First layer:

Input: S2 → 2D.Output: SO(3)→ 3D.The output is indexed by an entry in SO(3)

An extra dimension modeling the rotation

Movement over S2: 2 dofRotation around the position x: 1 dofDifferent from [2], which ”restricts the filter to be circularly symmetricabout the Z axis.”

CNNs on spherical images (higher layers)

(f ∗ g)(R) =

∫SO(3)

f (Q)g(R−1(Q))dQ

R,Q : (α, β, γ) ∈ SO(3)

Outline

4 Implementation

Transformation of the filter and the feature map

](t) = ψ(g−1t)[

Lg f](t) = f (g−1t)

Equivariance properties of CNNs

φ(Tgx) = T ′gφ(x).

transforming an input x by a transformation (e.g. translation) g(forming Tgx) and then passing it through the learned map φ shouldgive the same result as first mapping x through φ and thentransforming the representation.

Planar CNN is equivariant to translations.

([LT f ] ∗ ψ) = LT (f ∗ ψ)f: e.g. earlier CNN layers

Planar CNN is equivariant to translations.T (t) = t + u; T−1(x) = x − udT (t) = d(t + u) = dt

LT (f ∗ ψ)(x) = (f ∗ ψ)(T−1x) = (f ∗ ψ)(x − u)

f (y)ψ((x − u)− y)dy

f (y)ψ(x − (u + y))dy

{substitute : v = u + y} =

f (v − u)ψ(x − v)dv

f (T−1v))ψ(x − v)dv

[LT f ](v))ψ(x − v)dv

= ([LT f ] ∗ ψ)(x)Credit: Taco S. Cohen, Mario Geiger, Jonas Kohler, Max Welling (shortinst)Spherical CNNs Presenter: Fuwen Tan https://qdata.github.io/deep2Read 14 / 32

Outline

4 Implementation

Spherical CNN is rotation-equivariant

Spherical CNN is equivariant to rotations.

([LQ f ] ∗ ψ) = LQ(f ∗ ψ)Requirement:

dy is the invariant measure on S2

dQ is the invariant measure on SO(3)dRy = dy ; dRQ = dQ∫S2 θ(Ry)dy =

∫S2 θ(Ry)d(Ry) =

∫S2 θ(y)dy

guarantee by the parameterization (appendix A)

(f ∗ ψ)(x) =

(f ∗ ψ)(R) =

∫SO(3)

f (Q)ψ(R−1(Q))dQ

Proof (appendix B)

([LQ f ] ∗ ψ)(R) =

f (Q−1y)ψ(R−1y)dy

{substitute : y = Qt} =

f (t)ψ(R−1Qt)d(Qt)

f (t)ψ((Q−1R)−1t)d(t)

= (f ∗ ψ)(Q−1R)

=[LQ(f ∗ ψ)

GFFT defined on S2 and SO(3)

SO(3): Wigner D-function

S2: spherical harmonics

Outline

4 Implementation

Equivariance error

∆ =1

n∑i=1

std(LRi(Φ(fi ))− Φ(LRi

std(Φ(fi ))

Φ : spherical CNN layers with randomly initialized filters

fi ,Ri := randomly chosen features (with channel K=10) and rotations

n = 500

10−6no act. & 1 layer no act. & res. of 30

0 10 20 30

resolution

ReLU & 1 layer

1 5 10

layers

ReLU & res. of 30

Figure: ∆ as a function of the resolutionand the number of layers.

Outline

4 Implementation

MNIST on the sphere

Dataset 1 (NR): projected on the northern hemisphere

Dataset 2 (R): projected on the northern hemisphere and thenrandomly rotated

Planar images for baseline methods:

stereographic projection

Figure: Stereographic projection.

Results

Baseline: conv-ReLU-conv-ReLU-FC

kernel: 5 x 5channels: 32, 64, 10

Spherical CNN: S2conv-ReLU-SO(3)conv-ReLU-FC

bandwidth: 30, 10, 6channels: 20, 40, 10

NR / NR R / R NR / R

planar 0.98 0.23 0.11spherical 0.96 0.95 0.94

Table: Test accuracy for the networks evaluated on the spherical MNIST dataset.Here R = rotated, NR = non-rotated and X / Y denotes, that the network wastrained on X and evaluated on Y.

Outline

4 Implementation

3D recognition

SHREC17 task [3]Training data: 51300 non-aligned 3D modelsClassification: 55 categories

RepresentationRay casting on the surface and its convex hullchannels: 6 ((length, cos, sin) x 2)

ray castingfrom the sphere to the origin

distance sphere-impact

normal at impact

Figure: The ray is cast from the surface of the sphere towards the origin. Thefirst intersection with the model gives the values of the signal. The two images ofthe right represent two spherical signals in (α, β) coordinates. They containrespectively the distance from the sphere and the cosine of the ray with thenormal of the model. The red dot corresponds to the pixel set by the red line.

Results

S2conv-BN-ReLU (50 features)2 x (SO(3)conv-BN-ReLU) (70/350 features)max-pooling-BN-FCbandwidths: 128, 32, 22, 7

Method P@N R@N F1@N mAP NDCG

Tatsuma ReVGG 0.705 0.769 0.719 0.696 0.783Furuya DLAN 0.814 0.683 0.706 0.656 0.754SHREC16-Bai GIFT 0.678 0.667 0.661 0.607 0.735Deng CM-VGG5-6DB 0.412 0.706 0.472 0.524 0.624

Ours 0.701 (3rd) 0.711 (2nd) 0.699 (3rd) 0.676 (2nd) 0.756 (2nd)

Table: Results and best competing methods for the SHREC17 competition.

Outline

4 Implementation

Molecular energy regression

QM7 task

Input: for each molecule, positions pi and charges zi of the atomsN = 23 atoms of T = 5 types (H, C, N, O, S) for each moleculeOutput: atomization energy of the molecule (scalar)

Representation as a spherical signal

A sphere Si around pi

Uniform radius such that no intersections among spheres

For each possible z and for each point x ∈ S2:

Uz(x) =∑

j 6=i,zj=zzi ·z|x−pi |

For each atom: a T channel spherical function

ResNet block

S2SO(3)con − BN − ReLU − SO(3)conv − BN

Shared weights for all atoms: N x F feature maps

To achieve permutation invariance:

Embedding: MLP φSum poolingRegression: MLP ψ

Results

Take-home messages

One of the best papers in ICLR 2018

Potential applications on omnidirectional vision (e.g. for AR/VR)

Potential extensions to more transformation groups (e.g. SE(3))

Taco Cohen and Max Welling.

Group equivariant convolutional networks.

In International Conference on Machine Learning (ICML), pages 2990–2999.PMLR, 20–22 Jun 2016.

J. R. Driscoll and D. M. Healy.

Computing fourier transforms and convolutions on the 2-sphere.

Adv. Appl. Math., 15(2):202–250, June 1994.

M. Savva, F. Yu, Hao Su, M. Aono, B. Chen, D. Cohen-Or, W. Deng, HangSu, S. Bai, X. Bai, N. Fish, J. Han, E. Kalogerakis, E. G. Learned-Miller,Y. Li, M. Liao, S. Maji, A. Tatsuma, Y. Wang, N. Zhang, and Z. Zhou.

Large-scale 3d shape retrieval from shapenet core55.

In Proceedings of the Eurographics 2016 Workshop on 3D Object Retrieval,3DOR ’16, pages 89–98, Goslar Germany, Germany, 2016. EurographicsAssociation.

Spherical CNNs · 2020-01-26 · Rotated MNIST on the sphere Recognition of 3D shapes Prediction of...

Documents

Mnist Solution with Wolfram Mathematica

Flow mechanisms in high pressure gas atomization water sprays has been applied to gas atomiza- which is more specific to the gas atomization of tion of molten ... atomization practice,

Spectrally-normalized margin bounds for neural …0 cifar cifar random mnist (a)mnist iseasierthancifar10. 0 cifar cifar random mnist random (b)Randommnist isashardasrandomcifar10!

From Zero to CNNs to Brains - CBMM...Vision Barbu,Mayo,Conwell (MIT,Harvard) 0 !CNNs Brains August12,2020

Improved Atomization via a Mechanical Atomizer with ... · The atomization of the liquid is accomplished via various atomizer designs. The current designs for liquid atomization may

Atomization and Flames in LOX H and LOX CH -Spray …elib.dlr.de/53113/1/RP_4676.pdf · Atomization and Flames in LOX=H 2- ... under a different atomization regime as compared to

Minimizing Contamination in Titanium Atomizationtitanium.scholarlab.com/customer/titanium/resources/tieurope2014/... · Powder Production via Gas Atomization Gas Atomization: Flowing

Echo State Networks-based Reservoir Computing for MNIST

Zeoli Thesis doc of atomization

Towards Life Long Learning: Multimodal Learning of MNIST ... · Towards Life Long Learning: Multimodal Learning of MNIST Handwritten Digits Eli Sheppard y, Hagen Lehmann , G. Rajendran

Workshop on Electrostatic Atomization of Electrically Insulating Liquids : Principles and Applications Applications of Electrostatic Atomization in Industry

Atomization of viscous and non-newtonian liquids by a ...depts.washington.edu/fluidlab/Papers/Atomization...Atomization of viscous and non-newtonian liquids by a coaxial, high-speed

Fluid Atomization Innovation

Three ingredients of deep learning: LeNet-5, MNIST, and

CNNS Brochure

Shiou-Shiou Deng This homework only uses the MNIST …gchen/Math285S16/HW1-Shioushiou.pdf · This homework only uses the MNIST Digits dataset (both training and test data). ... 8

Open Compound Domain Adaptation - ICSI | ICSIstellayu... · Simulation Source Overcast Snowy Target Open Unknown Domains SVHN §ymNum USPS MNIST MNIST-M Source Compound Targets instance-wise

E(2)-Equivariant Steerable CNNs

MNIST and machine learning - presentation

2014.01.18 Atomization Model