35
Black Box Linear Algebra An Introduction to Wiedemann’s Approach William J. Turner Department of Mathematics & Computer Science Wabash College

Black Box Linear Algebra - Wabash Collegepersweb.wabash.edu/facstaff/turnerw/Presentations/ccs-2004.pdf · Black Box Linear Algebra An Introduction to Wiedemann’s Approach William

Embed Size (px)

Citation preview

Black Box Linear AlgebraAn Introduction to Wiedemann’s Approach

William J. Turner

Department of Mathematics & Computer Science

Wabash College

Overview

Symbolic computation

Black box matrix model

Wiedemann’s approach

Nonsingular systems

Singular systems

Rank

Determinant

Other work

Black Box Linear Algebra – p.1/32

Symbolic Computation

Superset of computer algebra

Different from numerical analysis

Symbols or exact arithmetic

Not floating point numbers (numerical analysis)

No round off errors

Algorithms may not be compatible

Black Box Linear Algebra – p.2/32

. . . vs. Numerical Analysis

Numerical Analysis

Find approximation quickly

May never find exact solution

Symbolic Computation

Find exact solution quickly

May never approximate

Black Box Linear Algebra – p.3/32

Probabilistic Algorithms

Three types:

Monte Carlo: Always fast, probably correct

Las Vegas: Probably fast, always correct

Requires certificate

BPP: Bounded Probabalistic Polynomial Time

Probably fast, probably correct

Atlantic City?

Schwartz-Zippel Lemma

Probability randomly choose root of polynomial

Black Box Linear Algebra – p.4/32

Solving Linear Systems

������������

� � � �

� � � �

� � �

� � � �

����������

� ��

�����������

� �

�����������

Black Box Linear Algebra – p.5/32

Gaussian Elimination

������������

� � � �

� � � �

� � �

� � � �

����������

� ��

�����������

�����������

� � � ��

�����������

� �

�����������

Row echelon form and back substitution

Must know how matrix stored

Matrix changed in calculation

Sparse matrices may become dense

Black Box Linear Algebra – p.6/32

Black Box Matrix Model

� � �� � � � � � ��

� � �� � �

External view of matrix

Only matrix-vector products allowed

Independent of implementation

Implementations efficient in time or space

Not unique to symbolic computation

Can compute Krylov sequence

� � � � � �

Black Box Linear Algebra – p.7/32

Examples

Matrix storage time

Arbitrary matrix � � � � � �

Sparse matrix

( � nonzero entries)

� � � � � �

Hilbert matrix

(

� � ��� � � �

� ��� )

� � � � � �

Toeplitz matrix

� � � � � � �� � � � �� � �� � � �

Black Box Linear Algebra – p.8/32

Linearly Recurrent Sequences

Let

be a vector space over field

.

A sequence

� � � � �

� � � � �is linearly recurrent if and only if there exist � � �

and

����� � �� � �� �� � � �

such that for all

� �

��� �

� � � � � � �

Black Box Linear Algebra – p.9/32

Linearly Recurrent Sequences

Let

be a vector space over field

.

A sequence

� � � � �

� � � � �is linearly recurrent if and only if there exist � � �

and

����� � �� � �� �� � � �

such that for all

� �� � �

���

� �� �

� � � � �

Black Box Linear Algebra – p.9/32

Generating Polynomials

The polynomial

� �� � �� � � � � �

generates �

� � � �

�� �

� � � � �� �

� � � � � � � � � �

� �

is an

� �� �

-module

Black Box Linear Algebra – p.10/32

Minimal Generating Polynomial

If

� � � � � and � � � �� �

, then

��

� � � � � � � � � � � � � � � �

� � � � �� � � � � � � � �

is an ideal

� �� �

is a principal ideal domain

There exists a unique monic generator of minimal degree, the

minimal generating polynomial of sequence

Minimal polynomial divides all generating polynomials

Black Box Linear Algebra – p.11/32

Example: Fibonacci Numbers

� � �

� � � � �� �� �� � �� �� �� � �� �� � ��

Minimal polynomial is

� � � � � �

� � � � � � � � �

�� � � � � � �

� � also generates �

� � � � � � � � �

� � � � � � � � � � �

also generates �

� � � � � � � � � � � �

Skips first

elements of �

Black Box Linear Algebra – p.12/32

Matrix Power Sequence

� � � � � � � � � � � � � �

��� � ��� �

generates

� � � � �

Cayley-Hamilton Theorem

Let

� �

be minimal polynomial of

� � � � �

� � � �� � ��� �

Black Box Linear Algebra – p.13/32

Krylov Sequence

� � � � � � � � � � � � � � � �

� � � � � � � � � � � � �

� � � � � � � � � � � � ��� � � � � � � � � � � � � � �

Let

� �� �

be minimal polynomial of

� � � � � �

� �� � � � � � �� � ��� �

Black Box Linear Algebra – p.14/32

Solving �

Suppose � �

��� �� � � �

, �� � � �� � � �, and � � � � ���

� �

If � exists, then

� �� �

satisfies requirements

� � � � � � � ��� � � � �� � � � �

� � �

� �

�� � � � � � � �

� �

��� � � � � ��� �

� � �

� �

��� � � � � ��� �

is a solution

Black Box Linear Algebra – p.15/32

Nonsingular

��� � � � � � �

�� � ��� � ��� � � � � �

� �� � � � � � �

� � �

� �

��� � � � � ��� �

is the unique solution

Black Box Linear Algebra – p.16/32

Bilinear Projection Sequence

� � ��T

� � � � � � � �T

� � � � � �

� � � � � � � � �T

� � � � � �

� � � � � � � � � � � � � �� � � � � � �

�T

� � � � � � � �

Let

� �� �� be minimal polynomial of

��T

� � � � �

� �� ��

� � �� � � � � � ��� � ��� � If � chosen randomly from finite

� � � � �

, then

� �� ��

� � �� �

with

probability at least

� �� � � � �� �

� � �

Certificate:� �� �

� � � � � � � � �

Black Box Linear Algebra – p.17/32

Computing Minimal Polynomial

Must know

��� � � � �

Extended Euclidean Algorithm

� � �� � � � �� � � �

Inputs are

�� �� �

� ��

�� � � � � � � and

�� �� � � � �

Stop when

�� � � � � � � �� � � � ���

Minimal polynomial is reversal of � � ��

Berlekamp-Massey Algorithm

From coding theory

Interpolates elements of scalar sequence

Related to Extended Euclidean Algorithm

Black Box Linear Algebra – p.18/32

Berlekamp-Massey Algorithm

Input: Scalar sequence

� � �

� � � � �

with generator � with�� � ��

� �

Output: Minimal polynomial

of sequence

1:

� � � �

Initial guess

2: for � � � to

� � do

� �

generates � �� � ��� � �

3:

� � � � � � � � � � � � ��� � �4:

� � �� ��� � � � � ��� � � ��� � � � ��� � �

5: if

� � � � then

� �

does not generate � � � � � ��

6: update

to generate � � � � ���

7: end if

8: end for

Black Box Linear Algebra – p.19/32

Example: Fibonacci Numbers

� � �

� � � � �� �� �� � �� �� �� � �� �� � ��

� �

recursion relation

� � � � � �

� � � � � � � �

� � � � � � � � � �

� � � � � � � � � � �

� � � � � � � � � � �

� � � � � � � � � � �

� � � � � � � � � � �

� � � � � � � � � � �

Black Box Linear Algebra – p.20/32

Wiedemann’s Algorithm

� �� ��

� � �� � � � � � ��� � ��� �

��� � � � �� ��

� �� � � �� � ��� � � �

Only need

��T

� � � � � �

�� �

Require: nonsingular

� � �� � �

and

� � ��

Ensure: � � ��

such that

� � � �1:

� �

random vector in

� �

where�� �

2: use Berlekamp-Massey to compute

� �� �� �� � � � � � � � � � � �

Store only

� � � �

3:

� �

��� � � � � � �� �� � � � � � �

Black Box Linear Algebra – p.21/32

Back to original problem

�����������

� � � �

� � � �

� � �

� � � �

���������

� ��

����������

� �

����������

� � �

� ��

�����������

�����������

Black Box Linear Algebra – p.22/32

Back to original problem

��T � � � � � �

�� � � � �� �� �� �� � �� � �� � �� � � �

� �� �� � �� �

� � � � �

� ��

� �� � � ��

���������

������������

�����������

������������

� �

�����������

���������

��

�����������

� �

�����������

Black Box Linear Algebra – p.23/32

Singular

Kaltofen and Saunders (1991):

If

� � � �� � � � � � known

� � � leading principal minor nonzero

Randomly choose �

Solve � � � nonsingular system

�� � � �� � � � � �� � � �

�� is leading � � � submatrix of

� �

is first � rows of

Then � �� � � ��

�� � �� �

��

� uniformly samples solution manifold

Black Box Linear Algebra – p.24/32

Precondition Matrix

Want � � � leading principal minor of

� �

nonzero

� � � � �

or

� � � � �

Need efficient matrix-vector product

New linear system

� � � � ���

Black Box Linear Algebra – p.25/32

Generic Rank ProfileWiedemann (1986):

� � � � �

,

parameterized and can realize any permutation

Kaltofen and Saunders (1991):

� � � ��� � ��� ,

��� unit upper triangular Toeplitz and

��� unit lower

triangular Toeplitz

Chen et al. (2002); Turner (2002):

� � � � � � � � ,

� � ,

� � based on butterfly networks

0 1 0 0 1 1 10

0 0 0 01 1 1 1

0 1 101 0 0 1

0 001 1 1 01

Generic exchange matrix mixes inputs, moving linear independence

Turner (2002): Generalize butterfly networks to radix-

switches

Turner (2004): Toeplitz matrix switches

Students: Hankel matrix switches

Black Box Linear Algebra – p.26/32

Matrix Rank

Kaltofen and Saunders (1991):

If

� � � �� � � � � � (unknown)

Leading principal minors nonzero up to

��

� � �� � � �� � ��

Then � � �� � � � �� � with probability at least

� � � � �

� � �

Black Box Linear Algebra – p.27/32

Matrix Minimal Polynomial

If � chosen randomly, then

� �� � � � �

with probability at least

� �� � � � �

� � �

� �

�� � �

If � and � chosen randomly, then

� �� ��

� � �with probability at

least

� �� � � � �

� � �

� � �

� � �

Black Box Linear Algebra – p.28/32

Rank PreconditionersGeneric rank profile preconditioner and diagonal matrix

Chen et al. (2002):

� � � ��

� ���

���

�� and

��� unit lower triangular Toeplitz,

��� upper triangular

Toeplitz

Turner (2002, 2003): Relax slightly generic rank profile

Turner (2004):

� � � � �

�����

� . . .

��

����

Toeplitz

Turner (2004):� � � � �

Black Box Linear Algebra – p.29/32

Matrix Determinant

If

� �� ��� � ��� �

Then

�� � � � � � �

� � � � �

Black Box Linear Algebra – p.30/32

Determinant PreconditionersGeneric rank profile preconditioner and diagonal matrix

Kaltofen and Pan (1992):

� � � � � � �

� unit upper triangular Toeplitz and

� � lower triangular Toeplitz

Chen et al. (2002):

� � � ��

���

. . .�

��

��

Turner (2002, 2003):� � � �

�����������

� �

. . .. . .

� � � � �

����������

Black Box Linear Algebra – p.31/32

Other Work

Krylov methods

Sparse preconditioners for small fields

Blocked algorithms

Preconditioners with more structure

Black Box Linear Algebra – p.32/32

ReferencesE. R. BERLEKAMP (1968). Algebraic Coding Theory. McGraw-Hill, New York.

L. CHEN, W. EBERLY, E. KALTOFEN, B. D. SAUNDERS, W. J. TURNER, and G. VILLARD (2002). Efficient Matrix Preconditioners

for Black Box Linear Algebra. Linear Algebra and its Applications, 343-344: 119–146. Special issue on Infinite Systems

of Linear Equations Finitely Specified.

J. L. DORNSTETTER (1987). On the Equivalence Between Berlekamp’s and Euclid’s Algorithms. IEEE Transactions on

Information Theory, IT-33(3): 428–431.

J. VON ZUR GATHEN and J. GERHARD (1999). Modern Computer Algebra. Cambridge University Press, Cambridge.

E. KALTOFEN (1992). Efficient Solution of Sparse Linear Systems. Lecture notes, Computer Science Department, Rensselaer

Polytechnic Institute, Troy, N.Y.

E. KALTOFEN and V. PAN (1991). Processor Efficient Parallel Solution of Linear Systems over an Abstract Field. In Proceedings

of SPAA ’91 3rd Annual ACM Symposium on Parallel Algorithms and Architectures, 180–191. ACM Press, New York, New

York.

E. KALTOFEN and V. PAN (1992). Processor Efficient Parallel Solution of Linear Systems II: the Positive Characteristic and

Singular Cases. In Proceedings of the 33rd Annual Symposium on Foundations of Computer Science, 714–723. IEEE

Computer Society Press, Los Alamitos, California.

E. KALTOFEN and B. D. SAUNDERS (1991). On Wiedemann’s Method of Solving Sparse Linear Systems. In H. F. MATTSON,

T. MORA, and T. R. N. RAO (eds.), AAECC-9: Proceedings of the 1991 Applied Algebra, Algebraic Algorithms and Error-

Correcting Codes, International Conference, vol. 539 of Lecture Notes in Computer Science, 29–38. Springer Verlag,

Heidelberg, Germany.

J. L. MASSEY (1969). Shift-Register Synthesis and BCH Decoding. IEEE Transactions on Information Theory, IT-15(1):

122–127.

J. T. SCHWARTZ (1980). Fast Probabilistic Algorithms for Verification of Polynomial Identities. J. ACM, 27: 701–717.

W. J. TURNER (2002). Black Box Linear Algebra with the LinBox Library. Ph.D. thesis, North Carolina State University, Raleigh,

North Carolina.

W. J. TURNER (2003). Determinantal Divisors and Matrix Preconditioners. Submitted to Journal of Symbolic Computation.

W. J. TURNER (2004). Preconditioners for Singular Black Box Matrices. Work in progress.

D. H. WIEDEMANN (1986). Solving Sparse Linear Equations Over Finite Fields. IEEE Transactions on Information Theory,

IT-32(1): 54–62.

R. ZIPPEL (1979). Probabilistic Algorithms for Sparse Polynomials. In E. W. NG (ed.), Symbolic and Algebraic Computation,

EUROSAM ’79, An International Symposiumon Symbolic and Algebraic Computation, Marseille, France, June 1979,

Proceedings, vol. 72 of Lecture Notes in Computer Science, 216–226. Springer Verlag, Heidelberg, Germany.

R. ZIPPEL (1990). Interpolating Polynomials from Their Values. Journal of Symbolic Computation, 9(3): 375–403.

32-1