Old tricks for new data: Computing forgenomic evaluation in livestock

improvement schemes

Karin Meyer and Bruce Tier

Animal Genetics and Breeding Unit,University of New England,

Armidale NSW 2351, Australia

CSIRO Genome to Phenome Symposium, Brisbane, March 25-27, 2015

| Introduction

Genetic improvement of livestock

Select “better” parents of next generation– genetic potential to produce more (meat/milk/wool/. . .)

Estimate “breeding values” / transmitting ability– Phenotypes −→ genotypes– Well established

◦ Dairy cattle 50+ years

Best Linear Unbiased Prediction (BLUP)– Linear, mixed model (continuous traits)

– Fixed: systematic environmental effects (herd, year, . . .)

– Random: animals’ additive genetic effects

Solve system of mixed model equations (MME)– always too large & challenging

. . . computer too slow, insufficient RAM, no s/ware . . .

K. M. | 2 / 19

| Introduction

Objectives

“Computational carpentry”

- Review strategies for routine, large scale computing

- Examples of use in genetic & genomic evaluation

K. M. | 3 / 19

| Introduction

‘Tricks’ for mixed model computing

1 Exploit known structure of matrices & equations. . . fastest way to multiply by 0 or 1 is to avoid it. . . build only one triangle of symmetric matrices

2 Utilize transformations. . . make non-zero elements go away

3 Sparse matrix storage & manipulations. . . storing elements of 0 not necessary

4 Find ‘equivalent’ models !. . . do the same thing, fitting a different model which hasthe same expectations & dispersion structure

K. M. | 4 / 19

| Introduction

‘Tricks’ for mixed model computing

1 Exploit known structure of matrices & equations. . . fastest way to multiply by 0 or 1 is to avoid it. . . build only one triangle of symmetric matrices

2 Utilize transformations. . . make non-zero elements go away

3 Sparse matrix storage & manipulations. . . storing elements of 0 not necessary

4 Find ‘equivalent’ models !. . . do the same thing, fitting a different model which hasthe same expectations & dispersion structure

K. M. | 4 / 19

The mixed model frameworkHow can we use its known structure?

| Mixed model framework

Mixed model equations

y = X b + Z u + e

ObservationsFixed effectsRandom effectsResidualsDesign matrix - RandomDesign matrix - FixedRelationship matrix

with Var(e) = R & Var(u) = ΣΣΣ⊗ A

MME:�

X′R−1X X′R−1ZZ′R−1X Z′R−1Z + ΣΣΣ−1 ⊗ A−1

��

bu

�

=

�

X′R−1yZ′R−1y

�

K. M. | 5 / 19

| Mixed model framework | Exploiting structure

MME: What do we know? I

1 Sort data within animals −→ R is blockdiagonal– read & process records for one animal at a time

2 Design matrices have few non-zero elements– typically: single non-zero element per effect × trait

xi =�

a a2

Covar.

| · · · 1 · · ·FE 1

| · · · 1 · · ·FE 2

| · · ·�

row for record i

– accumulate contributions to X′R−1(···) and Z′R−1(···)indirectly

– scatter LHS to sparse storage when all records/animaldone

K. M. | 6 / 19

| Mixed model framework | Exploiting structure

MME: What do we know? II

3 Inverse of pedigree based relationship matrix is sparse– Need ΣΣΣ−1 ⊗ A−1

– Can set up A−1 directly from list of pedigrees– Non-zero off-diagonal elements: only linking animal i and

parents j and k

Sire j Dam k Animal i

Sire j 0.5 · · · 0.5 · · · −1...

. . ....

...Dam k 0.5 · · · 0.5 · · · −1

......

. . ....

Animal i −1 · · · −1 · · · 2

K. M. | 7 / 19

| Mixed model framework | Single step

MME in the genomic age“Single-step” (SS) genomic BLUP

– Jointly evaluate genotyped and non-genotyped animals– Genomic information −→ genomic relationship matrix– Appealing

◦ No prediction equations◦ Back to the black box - backsolve for SNP effects◦ “Simply” replace A−1 by H−1

−→ currently being implemented for Australian sheep and beef cattle

Partition genetic effects & relationship matrix1 Non-genotyped animals2 Genotyped animals: G22 genomic relationship matrix

H−1 = A−1 +

�

0 00 G−1

22 − A−122

�

=

�

H11 H12

H21 H22

�

G22 and H22 are dense matrices!

K. M. | 8 / 19

| Mixed model framework | Single step

MME in the genomic age“Single-step” (SS) genomic BLUP

– Jointly evaluate genotyped and non-genotyped animals– Genomic information −→ genomic relationship matrix– Appealing

◦ No prediction equations◦ Back to the black box - backsolve for SNP effects◦ “Simply” replace A−1 by H−1

−→ currently being implemented for Australian sheep and beef cattle

Partition genetic effects & relationship matrix1 Non-genotyped animals2 Genotyped animals: G22 genomic relationship matrix

H−1 = A−1 +

�

0 00 G−1

22 − A−122

�

=

�

H11 H12

H21 H22

�

G22 and H22 are dense matrices!K. M. | 8 / 19

| Mixed model framework | Single step

Digression: Dense matrix calculations

Sparse MME: few ‘canned’ routines that can be utilisedSoftware for dense matrices readily available

– open source– reference library– highly optimised & multi-thread versions very useful!

◦ ATLAS, OpenBLAS◦ Intel MKL (commercial)◦ CUDA_BLAS & clBLAS for GPU

BLASBasic linear algebra subprograms, 1979

- Matrix × vector, matrix ×matrix, rank update

- Allow for special structures

LAPACKLinear algebra package, 1992

- Cholesky factorisation

- Solve system of equations

- Matrix inversionK. M. | 9 / 19

| Mixed model framework | Single step

Challenges in computing H−1

Compute G22 = SS′/ν large matrix product– S is n×m (no. of genotyped animals × no. of SNPs)– mn(n + 1) FLOPS (one triangle)– if S large: blockwise, out-of-core algorithms

Compute A22 undemanding– solve A A−1 = I, use A−1 = L−1L−T , known structure of L−1

Invert both G22 and A22– Operation count ∝ n3

– concerns: computing time - but accuracy?

●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●

●●

● ●1 hour six coressingle core

0

2

4

6

0 20000 40000 60000 80000Dimension

Tim

e (

hr) Computing time for matrix inversion

Double precision calculationsIntel I7 @ 3.2Ghz; 64GB RAMUse MKL routines DPOTRF & DPOTRI

K. M. | 10 / 19

| Mixed model framework | Single step

Solving the (genomic) MMEIterative solver: Pre-conditioned conjugate gradient

– partial Cholesky decompositionIndustrial scale

– Too large to fit in core– ‘Iteration on Data’ type schemes

◦ recognize patterns, factor out common terms◦ fast read operations

SS genomic BLUP– group genotyped animals, treat separately– dense storage of diagonal blocks or parts– exploit multi-threaded BLAS/LAPACK libraries

Tier, B. and Graser, H.-U. (1991). Predicting breeding values using an implicitrepresentation of the mixed model equations for a multiple trait animal model. J. Anim.Breed. Genet., 108(1):81–88.

Tier, B. (1992). Factoring out common terms in the mixed model equations. J. Anim. Breed.Genet., 109(1):81–89.

K. M. | 11 / 19

Equivalent models“Any linear model can be expressed as many different linear

models, all of which yield identical first and second moments ofthe data vector. The models of such a set are defined as

linearly equivalent.. . .

Consequently, if computations are too burdensome for aparticular data set and a particular model, it may be possible to

find an equivalent model that is computationally feasible.”

Henderson, C. R. (1985). Equivalent linearmodels to reduce computations.J. Dairy Sci., 68(9):2267–2277.

| Equivalent models

Simple example for an equivalent modelMultivariate analysis−→ genetic covar. matrix has factor-analytic structure

y = Xb + Zu + e with Var(u) = ΣΣΣ⊗ A

and

ΣΣΣ = ΓΓΓΓΓΓ′+ ΨΨΨ

Equivalent: XFA

y = Xb + Z(ΓΓΓ⊗ I)Z?

c + Z s + e

→ more equations

→ sparser coefficient matrix

common effectsVar(c) = I⊗ A

specific effectsVar(s) = ΨΨΨ⊗ A

K. M. | 12 / 19

| Equivalent models | SS examples

Unsymmetric extended SS genomic BLUP

‘standard’

(univar.)

. . .Z′1Z1 +λA11 λA12

λA21 Z′2Z2 +λ�

A22 + G−122 − A−1

22

�

.u1u2

=

.Z′1yZ′2y

Equivalent model no inverse!

. . .Z′1Z1 +λA11 λA12 0 0

λA21 Z′2Z2 +λA22 λI −λI0 λI λ A22 0

0 λI 0 λ G22

.u1u2

−δδδ−γγγ

=

.Z′1yZ′2y00

Legarra, A. and Ducrocq, V. (2012). Computational strategies for national integration of phenotypic,genomic, and pedigree data in a single-step best linear unbiased prediction. J. Dairy Sci., 95(8):4629 –4645.

K. M. | 13 / 19

| Equivalent models | SS examples

Equivalent model with polygenic effects

. . .Z′1Z1 +λA11 λA12

λA21 Z′2Z2 +λ�

A22 + G−122 − A−1

22

�

.u1u2

=

.Z′1yZ′2y

‘standard’ SS

(univariate)

Split: u2 = W g + a2

Stepwise solution scheme

genotype indicators: -1, 0, 1additive genetic SNP effectsresidual polygenic effects

. . .Z′1Z1 λA12

λA21 Z′2Z2 +λA22 +λω1 A−122

.u1u2

=

.Z′1y

Z′2y +ω2λ A−122 Wg

�

B−1 +ω2W′ A−122 W

�

g = ω2W′ A−122 u2

Liu, Z., Goddard, M., Reinhardt, F., and Reents, R. (2014). A single-step genomic model with directestimation of marker effects. J. Dairy Sci., 97(9):5833 – 5850.

K. M. | 14 / 19

| Equivalent models | PC model

Equivalent model for multi-trait analyses

Mixed model for q traits

y = Xb + Zu + e with Var(u) = ΣΣΣ⊗ H

– ΣΣΣ−1 ⊗ H−1 added to coefficient matrix in MME– add q2 coefficients for each non-zero element of H−1

Equivalent model

y = Xb + Z(Q⊗ I)Z?

(Q−1 ⊗ I)u

u?

+ e

with Var(u?) = Q−1ΣΣΣQ−T ⊗ H = ΛΛΛ⊗ H

– Choose Q so that ΛΛΛ is diagonal!,→ each non-zero element of H−1 adds only q coefficients,→ important in genomic age where H−1 has dense parts,→ trade-off: Z? is denser than Z

K. M. | 15 / 19

| Equivalent models | PC model

Equivalent model for multi-trait analyses

Mixed model for q traits

y = Xb + Zu + e with Var(u) = ΣΣΣ⊗ H

– ΣΣΣ−1 ⊗ H−1 added to coefficient matrix in MME– add q2 coefficients for each non-zero element of H−1

Equivalent model

y = Xb + Z(Q⊗ I)Z?

(Q−1 ⊗ I)u

u?

+ e

with Var(u?) = Q−1ΣΣΣQ−T ⊗ H = ΛΛΛ⊗ H

– Choose Q so that ΛΛΛ is diagonal!,→ each non-zero element of H−1 adds only q coefficients,→ important in genomic age where H−1 has dense parts,→ trade-off: Z? is denser than Z

K. M. | 15 / 19

| Equivalent models | PC model

Equivalent model for multi-trait analyses II

Suitable transformations−→ Based on eigen-decomposition ΣΣΣ = EΛΛΛE′

◦ Matrix of eigen-vectors: Q = E −→ Var(u?) = ΛΛΛ⊗ H

◦ Matrix of ‘factor loadings’: Q = EΛΛΛ12 −→ Var(u?) = I⊗ H

◦ . . . rotate: triangular form Q = EΛΛΛ12 T = L with T T′ = I

−→ reduce non-zero elements in Q from q2 to q(q + 1)/2

Beyond equivalent: dimension reduction– elements of u? are principal component (PC) scores

◦ in descending order of λi −→ variance explained– ignore PCs with negligible eigenvalues −→ no info

◦ truncate Q to first r ≤ q columns

– decrease computational requirements further

Kirkpatrick, M. and Meyer, K. (2004). Direct estimation of genetic principal components:Simplified analysis of complex phenotypes. Genetics, 168:2295–2306.

K. M. | 16 / 19

| Equivalent models | PC model

Equivalent model for multi-trait analyses II

Suitable transformations−→ Based on eigen-decomposition ΣΣΣ = EΛΛΛE′

◦ Matrix of eigen-vectors: Q = E −→ Var(u?) = ΛΛΛ⊗ H

◦ Matrix of ‘factor loadings’: Q = EΛΛΛ12 −→ Var(u?) = I⊗ H

◦ . . . rotate: triangular form Q = EΛΛΛ12 T = L with T T′ = I

−→ reduce non-zero elements in Q from q2 to q(q + 1)/2

Beyond equivalent: dimension reduction– elements of u? are principal component (PC) scores

◦ in descending order of λi −→ variance explained– ignore PCs with negligible eigenvalues −→ no info

◦ truncate Q to first r ≤ q columns

– decrease computational requirements further

Kirkpatrick, M. and Meyer, K. (2004). Direct estimation of genetic principal components:Simplified analysis of complex phenotypes. Genetics, 168:2295–2306.

K. M. | 16 / 19

Simple transformation cansubstantially reduce

computationalrequirements

| Equivalent models | PC model

Equivalent model for multi-trait analyses II

Suitable transformations−→ Based on eigen-decomposition ΣΣΣ = EΛΛΛE′

◦ Matrix of eigen-vectors: Q = E −→ Var(u?) = ΛΛΛ⊗ H

◦ Matrix of ‘factor loadings’: Q = EΛΛΛ12 −→ Var(u?) = I⊗ H

◦ . . . rotate: triangular form Q = EΛΛΛ12 T = L with T T′ = I

−→ reduce non-zero elements in Q from q2 to q(q + 1)/2

Beyond equivalent: dimension reduction– elements of u? are principal component (PC) scores

◦ in descending order of λi −→ variance explained– ignore PCs with negligible eigenvalues −→ no info

◦ truncate Q to first r ≤ q columns

– decrease computational requirements further

Kirkpatrick, M. and Meyer, K. (2004). Direct estimation of genetic principal components:Simplified analysis of complex phenotypes. Genetics, 168:2295–2306.

K. M. | 16 / 19

| Equivalent models | PC model

Application: SS genomic BLUP for sheep10.5 million records, q = 10 traits (LAMBPLAN terminal sires)

1,995,755 animals (inc. parents w/o records)

,→ 10,698 genotyped 0.5%

,→ G22 computed from 48,599 SNPNo. non-zero elem.s in inverse relationship matrix (triangle)

– Pedigree based A−1: 6,584,393– Pedigree + genomic H−1: 63,793,942

Other effects in model– contemporary groups– dams’ perm. environ. effects (6 traits, 653,067 levels)

– genetic groups (93 levels, fitted explicitly)

Intel Xeon CPU E5-2630– 16 cores @2.4Ghz, 20MB cache size– 256GB of RAM

K. M. | 17 / 19

| Equivalent models | PC model

Computing requirements for multi-trait SSNo. equations (Million)

PC6

PC7

PC8

PC9

PC10

MV10

0 5 10 15 20 25

15

18

21

25

28

100

No. non−zero elements (Billion)

PC6

PC7

PC8

PC9

PC10

MV10

0 2 4 6

Memory required (Gigabytes)

PC6

PC7

PC8

PC9

PC10

MV10

0 25 50 75 100

16

21

23

25

27

100

sequential

parallel

Computing time (Hours)

PC6

PC7

PC8

PC9

PC10

MV10

0 5 10 15

K. M. | 18 / 19

| Finale

Conclusions

Use of genomic information will increase computationaldemands in genetic evaluation schemes multi-fold∝ no. of non-zero elements in coefficient matrix

Good toolkit available paired with modern hardware– mixed model ‘trickery’– canned software, parallel computing– large RAM, multiple cores, GPU

Open conceptual challenges– SS genomic BLUP only 1st step– better way to derive/exploit genomic relationships→ LD? haplotypes? recursion to founders?

K. M. | 19 / 19