1

Workshop 4, Linz, RICAM, 12-16.12.2011

Advances in tensor numerical methods for

parameter-dependent and stochastic PDEs

Boris N. Khoromskij

http://personal-homepages.mis.mpg.de/bokh

Max-Planck-Institute for

Mathematics in the Sciences

Leipzig

Main references B. Khoromskij, Workshop 4, Linz, 12-16.12.11 2

B.N. Khoromskij, and Ch. Schwab, Tensor-Structured Galerkin

Approximation of Parametric and Stochastic Elliptic PDEs. SIAM J. Sci.

Comp., 33(1), 2011, 1-25.

B.N. Khoromskij, and I. Oseledets. Quantics-TT collocation

approximation of parameter-dependent and stochastic elliptic PDEs.

Comp. Meth. in Applied Math., 10(4):34-365, 2010.

S.V. Dolgov, V. Kazeev, and B.N. Khoromskij. Fast tensor-structured

solution of 1D elliptic SPDEs. Preprint MPI MiS, Leipzig 2011 (in

preparation).

B.N. Khoromskij. Introduction to Tensor Numerical Methods in Scientific

Computing. Lecture Notes, Preprint 06-2011, University of Zuerich,

Institute of Mathematics, 2011, pp 1 - 238.

http://www.math.uzh.ch/fileadmin/math/preprints/06 11.pdf

Tensor numerical methods in higher dim. B. Khoromskij, Workshop 4, Linz, 12-16.12.11 3

Numerical multilinear algebra.

Low-parametric separable approximation of multivariate

functions (formatted data compression).

High-dimensional integration.

Tensor representation of matrices and other transforms

(FFT, convolution, Green’s functions, FWT)

Identification of tensor structures:

tensor format ⇔ physical entities.

Model reduction via “projection” onto tensor-structured

manifold (low-parametric formats).

Tensor-truncated iterative/direct solvers for steady-state

and time dependent PDEs in higher dimension.

Motivations, challenges, recent progress B. Khoromskij, Workshop 4, Linz, 12-16.12.11 4

1. Modern Applications:

Large 3D problems: Green’s functions, Fourier transform, convolution.

Molecular systems: quantum molecular dynamics, DMRG in quant. chem.

PDEs in Rd: quantum computing, stochastic PDEs, atmospheric model.

Data compression: machine learning, data mining, image processing.

2. ”Curse of dimensionality”:

O(Nd)-meth. over N ×N × ...×N︸ ︷︷ ︸

d

grids (linear in volume size Nd).

3. O(dN)-complexity methods via separation of variables:

Tensor numerical methods to represent d-variate functions, operators,

and for solving equations on rank-structured low-parametric manifolds.

4. Super-compressed O(d logN)-represent. (log-volume):

Breaking-through idea: Quantized-TT (QTT) approx., Nd → O(d logN).

High-dimensional PDEs B. Khoromskij, Workshop 4, Linz, 12-16.12.11 5

Elliptic (parameter-dependent) BVP: Find u ∈ H10 (Ω), s.t.,

Hu := − div (a gradu) + V u = F in Ω ∈ Rd.

Elliptic EVP: Find a pair (λ, u) ∈ R×H10 (Ω), s.t., 〈u, u〉 = 1,

Hu = λu in Ω ∈ Rd,

u = 0 on ∂Ω.

Parabolic eq. (σ ∈ 1, i): Find u : Rd × (0,∞) → R, s.t.

u(x, 0) ∈ H2(Rd) : σ∂u

∂t+Hu = 0, H = −∆d + V (x1, ..., xd).

Tensor methods adapt gainfully to main challenges:

High spacial dimension: Ω = (−b, b)d ∈ Rd (d = 2, 3, ..., 100, ...).

Multiparametric eq.: a(y, x), u(y, x), y ∈ RM (M = 1, 2, ..., 100, ...,∞).

Limitations: ”curse of ranks“ = ”strong entanglements”.

Additive dim. splitting: Canonical (CP) format B. Khoromskij, Workshop 4, Linz, 12-16.12.11 6

Discretization in tensor-product Hilbert space of N-d tensors,

V = [V (i1, ..., id)] ∈ Vn = V1 ⊗ ....⊗ Vd ≡ Rn1×···×nd, nk = N .

Canonical rank-R tensors, CR = CR(Vn): V ∈ CR(Vn) if

V (i1, . . . , id) =

R∑

α=1

V1(i1, α) . . . Vd(id, α), Vk(·, α) ∈ Vk = Rnk .

V =R∑

α=1

V1(α)⊗ . . .⊗ Vd(α).

Storage: dRN , but CR is non-closed set in Vn!

d = 2: rank-R matrices. Visualizing canonical model, d = 3.

+

b

A

1b

V V V

V V V

V V V

+= ...+

1

1 2

2

2

r

r

r

(1) (1) (1)

(2) (2) (2)

21

(3) (3) (3)

rb

The Tucker format B. Khoromskij, Workshop 4, Linz, 12-16.12.11 7

Rank r = [r1, . . . , rd] Tucker tensors, Tr = Tr(Vn):

V = [V (i1, ..., id)] ∈ Tr(Vn) if [De Lathawer et. al. ’2000]

V (i1, . . . , id) =

r∑

α1,...,αd=1

G(α1, . . . , αd)V1(i1, α1) . . . Vd(id, αd).

Storage: drN + rd, r = max rℓ ≪ N (good for moderate d).

=

I 2

I 1

I 3

A B

I 1

r 2

r 1

I 2

I 3

r 3

V

V

V

(1)

(2)

(3)

MPS-type dim. splitting: TT/TC formats B. Khoromskij, Workshop 4, Linz, 12-16.12.11 8

In quantum physics: A matrix product states (MPS) representation of

slightly entangled systems [White ’92; Wang, Thoss ’03; Vidal ’04; Cirac ’06, ...].

Def. (Tensor Train/Chain format), (TT/TC[r]).

Given J := ×dℓ=1Jℓ, Jℓ = 1, ..., rℓ, J0 = Jd. V ∈ TC[r] ⊂ Vn if V is a

contracted product of tri-tensors in RJℓ−1×Iℓ×Jℓ over J ,

V (i1, ..., id) =∑

α∈J

G1(αd, i1, α1)G2(α1, i2, α2) · · ·Gd(αd−1, id, αd)

≡ G1(i1)G2(i2)...Gd(id),

where Gk(ik) is a rk−1 × rk matrix, 1 ≤ ik ≤ nk.

Recently rediscovered in numerical analysis:

Hierarchical dim. splitting (HDS), O(drlog dN) storage: [BNK ’06].

Hierarchical Tucker (HT) [Hackbusch, Kuhn ’09]

J0 = Jd = 1 (open b.c.): Tensor train (TT), [Oseledets, Tyrtyshnikov ’09].

J0 = Jd 6= 1 (periodic b.c.): Tensor chain (TC), [BNK ’09].

Benefits and limitations of the TT format B. Khoromskij, Workshop 4, Linz, 12-16.12.11 9

Storage: dr2N , n = (N, ...,N).

Rank bound: rℓ ≤ rankTT (V) := rank(V[ℓ]) ≤ rankCP (V).

V[ℓ] := [V (i1, ..., iℓ; iℓ+1, ..., id)] is the ℓ-mode TT unfolding matrix.

Quasi-optimal TT[r]-approx. of V ∈ Vn (robust QR/SVD),

minT∈TT [r]

‖V −T‖F ≤ (∑

ℓ=1,...,d−1

ε2ℓ )1/2, εℓ = min

rankB≤rℓ‖V[ℓ] − B‖F .

Manifold of fixed TT-rank tensors – [R. Schneider, Holtz, Rohwedder ’10]

Multilinear matrix-vector algebra: O(dRr3N2) – curse of ranks ?

d ∼ 100, R ∼ r ∼ 102, N ∼ 103

Quantization + TT format = QTT: N → logN , Nd → d logN

N

r1

r1rr

2 2r3

d=6

r

N

N

3

r6

r5

6

r5 r4

r r4

d= log N = 3

F

N=23

Toward tensor networks: efficient and robust? B. Khoromskij, Workshop 4, Linz, 12-16.12.11 10

α

α α

i1α2 i2

1 i1α

αd id

1 d. . .

idαα

iαα1 i 1 α1 α1 i 2α2 α2 αd−1 d

α

α

α1 i 2

d−1

. . .

. . .

α i2 α id−1

α

αd−1i dαd

d d i 1α α1 1

α 2

α2d−1

Further generalizations:

[BNK - Chemometrics and ILS, ’11] Survey on recent advances.

Quantized data: TT tour of higher dimensions B. Khoromskij, Workshop 4, Linz, 12-16.12.11 11

Quantization (folding) of a vector/tensor to higher (virtual) dimension

⇒ highly compressed algebraic represent. of functions, Nd→O(d log2N).

Def. [BNK ’09] N = 2L, n = N⊗d. The dyadic folding of degree L = logN ,

(isometry) Fd,L : Vn,d → Qm,D , m = 2⊗D, D = dL,

reshapes X ∈ Vn,d to the quantized 2× 2× ...× 2︸ ︷︷ ︸

dL

-tensor in Qm,D.

d = 1: a vector X(N,1) = [X(i)]Ni=1, is reshaped to L-dim. tensor,

F1,L : X(N,1) → A(m,L) = [A(j)], A(j) := X(i), j = j1, ..., jL.

For fixed i, the Q-multi-index j is defined via binary coding,

i− 1 =∑L

ν=1(jν − 1)2ν−1, jν − 1 ∈ 0, 1.

General concept of quantized-TT (QTT) format + basic approximation

results in QTT format, [BNK ’09]

2L × 2L matrix reshapes to a (2× 2)⊗L Q-matrix, [Oseledets ’09]

How does the QTT format work? B. Khoromskij, Workshop 4, Linz, 12-16.12.11 12

Lem. [BNK ’09] QTT -representation (approx.) of funct. related tensors

N = 2L, L ∈ N, c, z ∈ C. Quantized exponential N-vector

X := czn−1Nn=1 ∈ CN ,

is the rank-1, 2× 2× ...× 2︸ ︷︷ ︸

L

-tensor A,

F2,L : X 7→ A = c⊗Lp=1

1

z2p−1

, A : 1, 2⊗L → C.

For ∀α ∈ C, the trigonometric N-vector

X := sin(αh(n− 1))Nn=1 ∈ CN , h = 1/(N − 1),

has explicit rank-2 QTT-representation: yp = α2L−pip, ip = 0, 1, p = 1, ..., L,

X 7→ [sin y1cos y1]⊗L−1p=2

cos yp −sin ypsin yp cos yp

⊗

cos yL

sin yL

∈ 0, 1⊗L,

Proof by induction in view of Hint: sin z = eiz−e−iz

2i= Im(eiz).

Why the QTT model reduction is efficient B. Khoromskij, Workshop 4, Linz, 12-16.12.11 13

Polynomial of degree m 7→ QTT-image has TT-rank m+ 1.

QTT-rank of the step function and Haar wavelet is 1 and 2, resp.

For Gaussian g(x) := e−x2/2p2 , x ∈ [−a, a],

rankQTT (G) ≤ ca

p

√

log(ε−1p

1 + a).

Proof. The Fourier transform of Gaussian + rank-2 QTT of cos-function∫ ∞

−∞e−x

2/2p2 cos(ωx)dx = pe−ω2p2/2.

Rem. For the QTT representation of a vector F = f(xn)Nn=1,

xn = (n− 1)h, N = 2L, apply f(xn) = f(y1 + ...+ yL), with

yp = hip2p−1, ip = 0, 1, p = 1, ..., L, n− 1 =

∑L

p=1ip2

p−1,

⇒ rankQTT (F) ≤ separation-rank(f(x+ y)).

Rem. Rank-2 FTT decomposition of f(x) = f1(x1) + f2(x2) + . . .+ fd(xd),

f(x) =(

f1(x1) 1)

1 0

f2(x2) 1

· · ·

1 0

fd−1(xd−1) 1

1

fd(xd)

.

Multivariate polynomials B. Khoromskij, Workshop 4, Linz, 12-16.12.11 14

Lem. [BNK, Oseledets ’09] (QTT map of multivariate polynomials)

A general homogeneous polynomial potential of q = (q1, . . . , qd) ∈ Rd,

V (q) =d∑

i1,...,im=1

a(i1, . . . , im)m∏

k=1

qik , rankTT (V ) ≤ C0d[m2].

Harmonic potential: QTT-ranks are bounded by 4,

V (q) =d∑

k=1

wkq2k, rankTT (V ) ≤ 2, rankQTT (V ) ≤ 4

For Henon-Heiles potential

V (q) =1

2

d∑

k=1

q2k + λ

d−1∑

k=1

(

q2kqk+1 − 1

3q3k

)

,

rankTT (V ) ≤ 3, rankQTT (V ) ≤ 4.

Storage: a QTT-image of funct. in RN⊗d

: O(dr2 logN), r- average rank.

Notice: The Henon-Heiles potential applies in molecular dynamics.

Numerics: QTT approx. of functional tensors B. Khoromskij, Workshop 4, Linz, 12-16.12.11 15

Average TT-rank: r2 =1

d

d∑

ℓ=1

rℓ−1rℓ, Storage ≤ 2dr2 logN.

Function-related N-vector: F = f((i− 12)h)Ni=1, h = b

N, ε = 10−6, b = 1

N \ r e−αx2, α = 0.1 ÷ 102

sin(αx)x

, α = 1 ÷ 102 1/x e−x/x x, x10, 10√x

212 3.1 5.6 4.2 3.8 1.9/2.6/3.9

214 2.9 5.5 4.2 3.8 1.9/2.5/3.9

216 2.8 5.4 4.2 5.3 1.9/2.4/3.9

N \ r 1/(x1 + x2) e−‖x‖ e−‖x‖2∆

−12 1, ε = 10−6, 10−7, 10−8

29 5.0 9.4 7.8 3.6

210 5.1 9.4 7.7 3.6

211 5.2 9.3 7.5 3.7

QTT representation of PES in high dim. B. Khoromskij, Workshop 4, Linz, 12-16.12.11 16

Compression of Henon-Heiles potential vs. dimension d ≤ 256.

Figure 1: EVP-solution and approximation timings for Henon-Heiles pot., L = 7, d ≤ 256, ε = 10−6

Range of dimensions 4 ≤ d ≤ 256, QTT-storage of V : ≤ 62.5 KB ≪ Nd.

Logarithmic complexity scaling, O(logN), in 1D grid-size N = 2L.

TT/QTT represent. of matrices (MPO) B. Khoromskij, Workshop 4, Linz, 12-16.12.11 17

Matrix product operators (MPO). A multi-way TT/QTT-matrix,

A : X := Rn1 × . . .× Rnd 7→ Rm1 × . . .× Rmd =: Y

A (i1, j1, . . . , id, jd) =

r1∑

α1=1

. . .

rd−1∑

αd−1=1

U1 (i1, j1, α1)U2 (α1, i2, j2, α2) · . . . ·

· UD−1 (αd−2, id−1, jd−1, αd−1)UD (αd−1, id, jd) ,

Uk(ik, jk) is a rk−1 × rk matrix, 1 ≤ ik ≤ nk, 1 ≤ jk ≤ mk.

Def. For A ∈ L(X → Y) and X ∈ X denote the vector TT ranks of the

matrix-by-vector product AX by r1 . . . rd−1.

The operator TT rank of A is defined by

maxk=1...d−1,

rankTT (X)=1

rk(AX).

k-th vector TT rank of A is the rank of its TT unfolding,

rk = rank(A[k]) (1 ≤ k ≤ d− 1), with the elements

A[k] (i1j1 . . . ikjk ; ik+1jk+1 . . . idjd) = A(i1, j1, . . . , id, jd).

TT/QTT represent. of elliptic operators B. Khoromskij, Workshop 4, Linz, 12-16.12.11 18

Example. d-dimensional FD Laplacian.

∆d = ∆1 ⊗ I ⊗ ...⊗ I + I ⊗∆1 ⊗ I...⊗ I + ...+ I ⊗ I...⊗∆1 ∈ RN⊗d×N⊗d

,

∆1 = tridiag−1, 2,−1 ∈ RN×N , I is the N ×N identity.

For the canonical/Tucker rank: rankCP (∆d) = d, rankTuck(∆d) = 2.

Explicit rank-2 TT representation,

∆d =[

∆1 I]

⋊⋉

I 0

∆1 I

⊗(d−2)

⋊⋉

I

∆1

.

[Kazeev, BNK ’10] Explicit QTT representation, rankQTT (∆1) = 3,

∆1 =[

I J ′ J]

⋊⋉

I J ′ J

J

J ′

⊗(d−2)

⋊⋉

2I − J − J ′

−J−J ′

.

I =

1 0

0 1

, J =

0 1

0 0

.

“⋊⋉” is a regular matrix product of block core matrices, blocks being multiplied by means of tensor product.

Tensor numerical methods: Main ingredients B. Khoromskij, Workshop 4, Linz, 12-16.12.11 19

1. Discret. in tensor-prod. Hilbert sp. Vn = Rn1×···×nd , nk = N .

2. MLA in the rank-r tensor formats S ⊂ Vn:

S ⊂ CR, Tr, TCR,r, TT/TC[r], QTT [r], r = [r1, . . . , rd].

Tensor truncation (projection), TS : S0 → S ⊂ S0 ⊂ Vn,

based on SVD + (R)HOSVD + ALS/DMRG + multigrid.

Scalar/Hadamard/contracted/convolution products on S.

3. S-tensor approximation of functions and operators.

4. Tensor-truncated solvers on low-parametric manifold S:

S-truncated preconditioned iteration or dynamics on S.

Direct minimization on S: ALS/DMRG in TT/QTT format.

Direct S-tensor solution operators via A−1, exp(tA), Green’s functions.

Parametric Elliptic Eqs.: Stochastic PDEs B. Khoromskij, Workshop 4, Linz, 12-16.12.11 20

Find uM ∈ L2(Γ)×H10 (D), s.t.

AuM (y, x) = f(x) in D, ∀y ∈ Γ,

uM (y, x) = 0 on ∂D, ∀y ∈ Γ,

A := − div (aM (y, x) grad) , f ∈ L2 (D) , D ∈ Rd, d = 1, 2, 3,

aM (y, x) is smooth in x ∈ D, y = (y1, ..., yM ) ∈ Γ := [−1, 1]M , M ≤ ∞.

Additive case (via the truncated Karhunen-Loeve expansion)

aM (y, x) := a0(x) +M∑

m=1

am(x)ym, am ∈ L∞(D), M → ∞.

Log-additive case

aM (y, x) := exp(a0(x) +M∑

m=1

am(x)ym) > 0.

Sparse stochastic Galerkin/collocation: [Babuska, Nobile, Tempone ’06-’10; Schwab el. ’07-’10]

Stochastic Galerkin, CR format, additive c.: [BNK, Ch. Schwab, SISC, ’11]

QTT, both additive and log-additive cases: [BNK, Oseledets, CMAM, ’10]

Stochastic collocation (additive case) B. Khoromskij, Workshop 4, Linz, 12-16.12.11 21

A parametric linear system, N - grid size in x (Galerkin-FEM, FD in x)

A(y)u(y) = f, f ∈ RN , u(y) ∈ RN , y ∈ Γ, (1)

A(y) = A0 +M∑

m=1

Amym, Am ∈ RN×N , parameter dependent matrix.

Collocation on n⊗M grid, n - grid size in y (uniform, Chebyshev, etc.)

y(k)m =: Γm ∈ [−1, 1], k = 1, . . . , n, Γyn =M⊗

m=1

Γm

⇒ Assembled large linear system

Au = f , u, f ∈ RN×n⊗M, A ∈ R(N×n⊗M )×(N×n⊗M ),

A = A0 × I × . . .× I +A1 ×D1 × I × . . .× I + . . .+ AM × I × . . .×DM ,

Dm, m = 1, . . . ,M , is n× n diagonal matrix with positions of collocation

points, y(k)m ∈ Γm, on the diagonal: rankCP (A) ≤M .

f = f × e× . . .× e, e = (1, ..., 1)T ∈ Rn.

Stochastic collocation (log-additive case) B. Khoromskij, Workshop 4, Linz, 12-16.12.11 22

In log-additive case the dependence on y is no longer affine.

Apply collocation to (1) ⇒ nM linear systems (p.w.l. FEM),

A(j1, . . . , jM )u(j1, . . . , jM ) = f, 1 ≤ jm ≤ n ⇒ Au = f .

A(i, j, y) =

∫

Db(y, x)

∂φi

∂x

∂φj

∂xdx, y ∈ Γyn, D = [0, 1].

A(i, i, y) = b(xi−1/2, y) + b(xi+1/2, y),

A(i− 1, i, y) = A(i, i− 1, y) = −b(xi−1/2, y),

for i = 1, ...,N , with

b(y, x) = eaM (y,x) = ea0(x)M∏

m=1

eam(x)ym , y ∈ Γyn.

There is still good low rank CP approximations of the form (and QTT)

A ≈R∑

k=1

M⊗

m=0

Amk, Amk ∈ R(M+1)×n.

Stochastic collocation (log-additive case) B. Khoromskij, Workshop 4, Linz, 12-16.12.11 23

Lem. 1D sPDE by p.w.l. FEM, the log-additive case, y ∈ Γyn:

rankClocA(i, i, y) ≤ 2, rankClocA(i, i− 1, y) = 1,

rankQTTlocA(i, j, y) ≤ 2, i, j ≤ N ,

rankCP (A) ≤ 4N .

Prof.

A(y) = Z(y) +D(y) + Z⊤(y), y ∈ Γyn.

D(y) is a diagonal of A, Z is the first subdiagonal,

D(y) =N∑

i=1

A(i, i, y)eie⊤i =

1

4(C1(y) + 2C2(y) + C3(y)),

C2(y) =

N∑

i=1

eie⊤i e

a0(xi)M∏

m=1

eam(xi)ym . (2)

C2(y), y ∈ Γyn, is NnM ×NnM diagonal matrix, and each summand in (2)

has tensor rank-1, ei - i-th identity vector.

QTT-ranks in variable ym, are equal to 1 (exponential function in ym).

I. Tensor-truncated preconditioned iteration B. Khoromskij, Workshop 4, Linz, 12-16.12.11 24

Parametric elliptic BVP on nonlinear manifold S:

A(y)u(y) = f ,

um+1 = um − B−1(Aum − f), um+1 := TS(um+1) ∈ S.

Assumptions:

u, f allow the low S-rank approximation,

A and B−1 are of low matrix S-rank,

A and B are spectral equivalent (close).

Good candidates for B−1:

(A) Shifted FD d-Laplacian inverse (∆d + aI)−1

∆d = ∆1 ⊗ IN ⊗ ...⊗ IN + ...+ IN ⊗ IN ...⊗∆1 ∈ RN⊗d×N⊗d

.

(B) A−1(y∗).

(C) Reciprocal preconditioner B−1 = ∆−1d A[1/a]∆

−1d .

II. Tensor-truncated preconditioned iteration B. Khoromskij, Workshop 4, Linz, 12-16.12.11 25

[BNK, Ch. Schwab ’10, SISC] Canonical format.

u(k+1) := u

(k) − ωB−1k

(Au(k) − f

), u

(k+1) = Tε(u(k+1)) → u,

Tε is the rank truncation operator preserving accuracy ε.

In additive case, a good choice of a (rank-1) preconditioner

B−10 = A−1

0 × I × . . .× I.

In log-additive case, adaptive preconditioner at iter. step k,

B−1k = A(y∗k)

−1 × I × . . .× I, y∗k = argminQTT (‖f − Au(k)‖).

Note: B0 corresponds to y∗ = 0.

Proven spectral equivalence, B0 ∼ A, in both cases.

Numerics: additive case, canonical format B. Khoromskij, Workshop 4, Linz, 12-16.12.11 26

[BNK, Ch. Schwab, ’11, SISC]

Preconditioned S-truncated iteration in (d+M)-dimensional parametric

space. Canonical format, M ≤ 100.

N⊗(M+d)-grid, d = 1, M = 20 (S = CR, B−1 := A(0)−1).

Variable coefficients with exponential decay (N = 63, R ≤ 5),

am(x) = 0.5 e−αmsin(mx), m = 1, 2, ....,M, x ∈ (0, π).

1 2 3 4 510

−4

10−3

10−2

10−1

Dim=20, alpha=1, rank=5, grid=63

rank

2−no

rm

1 2 3 4 5 6 710

−5

10−4

10−3

10−2

10−1

100

101

Dim=20, alpha=1, rank=5, grid=63

T−iter

Res

idua

l

Numerics: additive case, canonical, nonsmooth coef. B. Khoromskij, Workshop 4, Linz, 12-16.12.11 27

Smooth and random coefficient in y.

a(y, x) = a(y) := 1 +M∑

m=1

amym with γ = ‖a‖ℓ1 :=M∑

m=1

|am| < 1,

for the truncated sequence of (spatially homogeneous) coefficients

am = (1 +m)−α, (m = 1, ...,M) with algebraic decay rates α = 2, 3, 5

The zero order sPDE,

a(y)u(y) = f. (3)

Highly oscillating random coefficient

a(y) = 1 +

M∑

m=1

amymH(ym − cm(ym)),

the pwc function cm(ym), given by a random n-vector at [−1, 1].

H : R → −1, 1, H(x) = −1 for x < 0, and H(x) = 1 for x ≥ 0.

Numerics: additive case, canonical, nonsmooth coef. B. Khoromskij, Workshop 4, Linz, 12-16.12.11 28

1 2 3 4 510

−6

10−5

10−4

10−3

10−2

rank

2−no

rmDim=20, alpha=3, rank−max=5, grid=63

0 10 20 30 40 50 60 70−0.3

−0.2

−0.1

0

0.1

0.2

0.3

0.4

Figure 2: Approximation error vs. rank R (left) and the five canonical

vectors in variable y1 (right) for the solution of (3), M = 20.

r-convergence exponential, same as in the smooth case.

The canonical vectors are highly oscillating.

Matrix ranks: QTT-rank/both cases B. Khoromskij, Workshop 4, Linz, 12-16.12.11 29

[BNK, Oseledets, ’10, CMAM] Stratified 2D-dimensional sPDE in the two cases:

1. Polynomial decay: am(x) = 0.5(m+1)2

sinmx1, x1 ∈ [−π, π], m = 1, . . . ,M .

2. Exponential decay: am(x) = e−0.7m sinmx1, x1 ∈ [−π, π], m = 1, . . . ,M .

The parametric space is discretized on a uniform mesh in [−1, 1] with 2p

points in each spatial direction, p = 8.

M QTT-rank(10−7) QTT-rank(10−3)

5 33 11

10 43 21

20 51 23

40 50 25

Table 1: QTT-rank of the matrix, 2D SPDE, log-additive case,

exponential decay N = 128.

Sublinear dependence on M .

Numerics: QTT/log-additive B. Khoromskij, ILAS, Workshop 4, Linz, 12-16.12.11 30

Figure 3: The stratified 2D example with two different truncation parameters, 1-point preconditioner,

Left: Residue with iteration, Right: Ranks with iteration, Bottom: CPU, t = O(M).

d = 1: Rank bound on the solution B. Khoromskij, Workshop 4, Linz, 12-16.12.11 31

General assumption: there exists amin > 0, s.t.,

(A) amin ≤ a0(x) < ∞,

(B)

∣∣∣∣∣

M∑

m=1am(x)ym

∣∣∣∣∣≤ γamin with γ < 1, and for |ym| < 1.

Define the reassembled coefficients, bm(ym, x) = σma0(x) + am(x)ym, with

σm =‖am‖

∑Mm=1 ‖am‖

, (m = 1, ...,M).

Prop. [BNK ’11, CILS survey] Let d = 1, and assume ∇xv(x) ∈ C(D), with

v = −∆−1x f , ∇xuM (y, x) ∈ C(D) for all y ∈ Γ,

Then for ε-rank:

rankCloc (∇xuM ) ≤ C| log ε| (additive),

rankCloc (∇xuM ) = 1 (log − additive).

Complexity for the particular solution uM (y, x):

O(MNn| log ε|) (additive); O(MNn) (log-additive).

d = 1: Rank bound on the solution B. Khoromskij, Workshop 4, Linz, 12-16.12.11 32

Proof. ∇Tx (aM∇xuM −∇xv) = 0 ⇒ ∇xuM (y, x) = 1aM (y,x)

(C0 +∇xv(x)).

Then, in additive case, there exist ck, tk ∈ R>0, s.t.

∥∥∥∥∥∥

∇xuM (y, x)−K∑

k=−K

ck

M∏

m=1

e−tkbm(ym,x)(C0 +∇xv(x))

∥∥∥∥∥∥L∞

≤ Ce−βK/ logK ,

β, C > 0 do not depend on M and K.

Log-additive: rankCloc (1

aM (y,x)) = 1.

Remark on “deterministic” case, M = 0, pwl FEM.

Γ[a] =[

(a∇φi,∇φj)L2(D)

]

, fi = (f(x), φi)L2(D) i, j = 1, ...,N.

P2 = ∆−1h Γ[

1

a(x)]∆−1h ≈ Γ[a]−1, ∆h = Γ[1].

Lem. Let d = 1, then P2Γ[a] = I +R, where rank(R) = 1, R = ψηT .

[Dolgov, BNK, Oseledets, Tyrtyshnikov, LAA ’11]

d = 1: Direct QTT-solver vs. GMRES B. Khoromskij, Workshop 4, Linz, 12-16.12.11 33

Discrete analogy of Prop.:

Use of rank-1 preconditioner + GMRES,

P2Γ[a]u = P2f, P2Γ[a] = I + ψηT .

Direct solver

u =(I + ψηT

)−1P2f,

by using the Sherman-Morrison-Woodberry,

(I + ψηT

)−1= I −

ψηT

1 + ηTψ.

Γ[a]−1 =

(I −

ψηT

1 + ηTψ

)∆−1

h Γ[1/a]∆−1h .

Parametric case: rankQTT (∆−1h ) ≤ 5. Compute reciprocal

1/a(y, x) by Newton meth., QTT of(1 + ηTψ

)−1⇒ the

computable explicit solution operator with O(M logN logn)

QTT-complexity. [Dolgov, Kazeev, BNK ’11, in progress]

d = 1: Direct QTT-solver vs. GMRES B. Khoromskij, Workshop 4, Linz, 12-16.12.11 34

Parametric preconditioner and direct solver,

P2 = ∆−1h Γ[1/a(y)]∆−1

h .

∆h = I ⊗ · · · ⊗ I ⊗∆h, ∆−1h = I ⊗ · · · ⊗ I ⊗∆−1

h .

u = Γ[a(y)]−1f =

(

I − ψ(y)η(y)T

1 + η(y)Tψ(y)

)

P2f ,

Let v = P2f , ex be a vector of all ones and size N , then

u = v − (1

1 +N∑

i=1η(y, xi) · ψ(y, xi)

⊗ ex) ∗ ψ ∗((

N∑

i=1

η(y, xi) · v(y, xi))

⊗ ex

)

,

“∗” is a pointwise (Hadamard) product in both x and y,

“·” is a Hadamard product in y.

Crucial points:

– QTT approximation of a and the reciprocal, 1/a (Newton iteration).

– Hadamard multiplication in y variable (dimension n⊗M).

– Pointwise inverse of denominator, 1

1+N∑

i=1η(y,xi)·ψ(y,xi)

by Newton iter.

QTT of a, reciprocal 1/a, and denominator B. Khoromskij, Workshop 4, Linz, 12-16.12.11 35

Low QTT ranks of the coefficient tensor a at fixed spatial point:

Algorithm.

Set A = 0. For i = 1, ...,N

Compute QTT approximation of ai = a(xi, y),

Add the current component: A = A+ ai ⊗ ei,

Perform the compression A = Tε(A).

ei is the i-th identity vect., ai ⊗ ei is the comp. with the proper x index.

A in QTT: the Newton nonlinear iteration for f(X) = 1/X −A,

Xk+1 = Xk(2I −AXk), k = 0, 1, ...

The pointwise (Hadamard) QTT-multiplications and truncations:

Yk = Tε(XkAXk),Xk+1 = Tε(2Xk − Yk).

Works for the QTT-representation of denominator.

QTT of a, reciprocal 1/a, and denominator B. Khoromskij, Workshop 4, Linz, 12-16.12.11 36

The reciprocal 1/a in the additive case: use the quadrature approx.

1

z1 + ...+ zd≈

R∑

k=−R

ck

d∏

q=1

exp(−tkzq), tk = ekζ , ck = ζtk, ζ =π√R, (4)

∥∥∥∥∥∥

1

z1 + z2 + ...+ zd−

R∑

k=−R

ck

d∏

q=1

exp(−tkzq)

∥∥∥∥∥∥

= O(exp(−π√R)).

Fix x = xi, the additive coefficient turns into the form we need (use (4)):

a(xi, y) = α0 + α1y1 + ...+ αMyM , αm = am(xi).

Compute the log-additive coef. a by a sum of rank-1 components

N∑

i=1

exp(a0(xi))M∏

m=1

exp(am(xi)ym)⊗ ei.

Rem. rankQTT (exp(αy)) = 1 for any α. The rank of the sum ≤ N .

The reciprocal in the log-additive case is computed by a summation

N∑

i=1

exp(−a0(xi))M∏

m=1

exp(−am(xi)ym)⊗ ei.

Numerics: d = 1, “direct” sPDE solver vs. GMRES B. Khoromskij, Workshop 4, Linz, 12-16.12.11 37

ε = 10−5. CPU time in sec. Indication on log− log scaling.

• Polynomial decay: am(x) = 0.5(m+1)2

sinmx, x ∈ [−π, π], m = 1, . . . ,M .

Figure 4: u = (I + ψηT )−1v versus N,

n. M = 40. Additive case, polynomial decay.

Figure 5: TT-GMRES versus N, n. M =

40. Additive case, polynomial decay.

Numerics: d = 1, “direct” sPDE solver vs. GMRES B. Khoromskij, Workshop 4, Linz, 12-16.12.11 38

• Exponential decay: am(x) = e−0.7m sinmx, x ∈ [−π, π], m = 1, . . . ,M .

Figure 6: u = (I + ψηT )−1v vs. N, n.

M = 40. Additive case, exponential decay.

Figure 7: TT-GMRES vs. N, n. M =

40. Additive case, exponential decay.

Conclusions B. Khoromskij, Workshop 4, Linz, 12-16.12.11 39

C + QTT + Preconditioned tensor-truncated iteration:

unified approach to challenging probl. of numerical SPDEs.

– Separation rank estimates and analytic approximation

– Rank-structured tensor representation of the sytem matrix

– C/TT/QTT-structured preconditioners

– Fast and stable rank optimization MLA

– Separation of physical and stochastic variables ?

– Quantization in the physical variable ?

Fast direct QTT-solver for 1D sPDEs.

– Precomputing of a and 1/a by QTT-Newton iterations

– QTT representation of the rank-1 reciprocal preconditioner