Linear Algebra and MATLAB Tutorial

Jia-Bin HuangUniversity of Illinois, Urbana-Champaign

www.jiabinhuang.comjbhuang1@Illinois.edu

Today’s class• Introduction to MATLAB

• Linear algebra refresher

• Writing fast MATLAB code

Working with MATLAB

Learning by doing• Let’s work through an example!

• Download the example script here• Topics

• Basic types in Matlab• Operations on vectors and matrices• Control statements & vectorization• Saving/loading your work• Creating scripts or functions using m-files• Plotting• Working with images

Questions?

Linear algebra refresher• Linear algebra is the math tool du jour

• Compact notation• Convenient set operations• Used in all modern texts• Interfaces well with MATLAB, numpy, R, etc.

• We will use a lot of it!!

Slides credits: Paris Smaragdis and Karianne Bergen

Filtering, linear transformation-> extracting frequency sub-bands

Vector dot products-> measure patch similarities

Hybrid Image Image Quilting Gradient Domain Fusion

Solving linear systems-> solve for pixel values

Image-based Lighting

Solving linear systems -> solve for radiance map

Video Stitching and Processing

Solving linear systems -> solve for geometric transform

Scalars, Vectors, Matrices, Tensors

How will we see these?• Vector: an ordered collection of scalars (e.g., sounds)

• Matrix: a two-dimensional collection of scalars (e.g., images)

• Tensor: 3D signals (e.g., videos)

Element-wise operations• Addition/subtraction

• Multiplication (Hadamard product)

• No named operator for element-wise division• Just use Hadamard with inverted elements

c = a + b;c = a – b;

c = a.*b;

c = a./b;

Which division?Left Right

Array division C = ldivide(A, B);C = A.\B;

C = rdivide(A, B);C = A./B;

Matrix division C = mldivide(A, B);C = A/B;

C = mldivide(A, B);C = A\B;

EX: Array division

A = [1 2; 3, 4];B = [1,1; 1, 1];

C1 = A.\B;C2 = A./B;

C1 = 1.0000 0.5000 0.3333 0.2500

C2 = 1 2 3 4

EX: Matrix divisionX = A\B;-> the (least-square) solution to AX = BX = A/B;-> the (least-square) solution to XB = A

B/A = (A'\B')'

Transpose• Change rows to columns (and vice versa)

• Hermitian (conjugate transpose)• Notated as • Y = X’; % Hermitian transpose• Y = X.’; % Transpose (without conjugation)

Visualizing transposition• Mostly pointless for 1D signals

• Swap dimensions for 2D signals

Reshaping operators• The vec operator

• Unrolls elements column-wise• Useful for getting rid of matrices/tensors• B = A(:);

• The reshape operator• B = reshape(A, sz);• prod(sz) must be the same as numel(A)

EX: Reshape

A = 1:12; B = reshape(A,[3,4]); B = 1 4 7 10 2 5 8 11 3 6 9 12

Trace and diag• Matrix trace

• Sum of diagonal elements

• The diag operator EX: trace A = [1, 2; 3, 4]; t = trace(A); Q: What’s t?

EX: diag x = diag(A); D = diag(x); Q: What’s x and D?

Vector norm (how “big" a vector is)• Taxicab norm or Manhattan norm

• Euclidean norm

• Infinity norm

• P-norm

Unit circle (the set of all vectors of norm 1)

Which norm to use?• Say residual = (measured value) – (model estimate)

• : if you want to your measurement and estimate to match exactly for as many samples as possible

• : use this if large errors are bad, but small errors are ok.

• : use this if no error should exceed a prescribed value .

Vector-vector products• Inner product

• Shorthand for multiply and accumulate

• Outer product:

Matrix-vector product • Generalizing the dot product

• Linear combination of the columns of A

Matrix-matrix product C = ABDefinition: as inner products

As sum of outer productsAs a set of matrix-vector products• matrix and columns of

• rows of and matrix

Matrix products• Output rows == left matrix rows• Output columns == right matrix columns

Matrix multiplication properties• Associative:

• Distributive:

• NOT communitive:

Linear independence• Linear dependence: a set of vectors is linear dependent if there

exists a set of scalars with at least one such that

i.e., one of the vectors in the set can be written as a linear combination of one or more other vectors in the set.

• Linear independence: a set of vectors is linear independent if it is NOT linearly dependent.

Basis and dimension• Basis: a basis for a subspace is a linear independent set of vectors

that span • and are both bases that span • Not unique

• Dimension dim(): the number of linearly independent vectors in the basis for

Range and nullspace of a matrix • The range (column space, image) of a matrix

• Denoted by • The set of all linear combination of the columns of

• The nullspace (kernel) of a matrix • Denoted by • The set of vectors z such that

Rank of • Column rank of : dimension of

• Row rank of : dimension of

• Column rank == row rank

• Matrix is full rank if

System of linear equations• Ex: Find values that satisfy

Solution:• Step 1: write the system of linear equations as a matrix equation

• Step 2: Solve for

Mini-quiz 1• What’s the and for the following linear equations?

Mini-quiz 2• What’s the , and for the following linear equation?

Solving system of linear equations• Given and , find such that • Special case: a square matrix • The solution: • The matrix inverse exists and is unique if and only if

• is a squared matrix of full rank

• “Undoes” a matrix multiplication

• Y

Least square problems• What if ?

• No solution exists such that

• Least Squares problem: • Define residual • Find vector that minimizes

|

Least square problems• Decompose into components with

• and • .

• Since is in , the orthogonal complement of , the residual norm

which is minimized when and

Normal equations• The least squares solution occurs when , or equivalently

• Normal equations: • If has full column rank, then is invertible. • Thus has a unique solution

• x = A\b;• x = inv(A’*A)*A’*b; % same as backslash operator

Questions?

Writing Fast MATLAB Code

Using the Profiler• Helps uncover performance problems

• Timing functions: • tic, toc

• The following timings were measured on - CPU i5 1.7 GHz- 4 GB RAM

• http://www.mathworks.com/help/matlab/ref/profile.html

Pre-allocation Memory

3.3071 s

>> n = 1000;

2.1804 s2.5148 s

Reducing Memory Operations

>> x = 4;>> x(2) = 7;>> x(3) = 12;

>> x = zeros(3,1);>> x = 4;>> x(2) = 7;>> x(3) = 12;

Vectorization

2.1804 s 0.0157 s139x faster!

Using Vectorization• Appearance

• more like the mathematical expressions, easier to understand.

• Less Error Prone• Vectorized code is often shorter.• Fewer opportunities to introduce programming errors.

• Performance: • Often runs much faster than the corresponding code containing loops.

See http://www.mathworks.com/help/matlab/matlab_prog/vectorization.html

Binary Singleton Expansion Function• Make each column in A zero mean>> n1 = 5000;>> n2 = 10000;>> A = randn(n1, n2);

• See http://blogs.mathworks.com/loren/2008/08/04/comparing-repmat-and-bsxfun-performance/

0.2994 s 0.2251 s

Why bsxfun is faster than repmat?- bsxfun handles replication of the array

implicitly, thus avoid memory allocation- Bsxfun supports multi-thread

Loop, Vector and Boolean Indexing• Make odd entries in vector v zero• n = 1e6;

• See http://www.mathworks.com/help/matlab/learn_matlab/array-indexing.html• See Fast manipulation of multi-dimensional arrays in Matlab by Kevin Murphy

0.3772 s 0.0081 s 0.0130 s

Solving Linear Equation System

0.1620 s 0.0467 s

Dense and Sparse Matrices

• Dense: 16.1332 s• Sparse: 0.0040 s

More than 4000x faster!

Useful functions: sparse(), spdiags(), speye(), kron().0.6424 s 0.1157 s

Repeated solution of an equation system with the same matrix

3.0897 s 0.0739 s 41x faster!

Iterative Methods for Larger Problems• Iterative solvers in MATLAB:

• bicg, bicgstab, cgs, gmres, lsqr, minres, pcg, symmlq, qmr• [x,flag,relres,iter,resvec] = method(A,b,tol,maxit,M1,M2,x0)

• source: Writing Fast Matlab Code by Pascal Getreuer

Solving Ax = b when A is a Special Matrix• Circulant matrices

• Matrices corresponding to cyclic convolution Ax = conv(h, x) are diagonalized in the Fourier domain>> x = ifft( fft(b)./fft(h) );

• Triangular and banded• Efficiently solved by sparse LU factorization>> [L,U] = lu(sparse(A)); >> x = U\(L\b);

• Poisson problems• See http://www.cs.berkeley.edu/~demmel/cs267/lecture25/lecture25.html

In-place Computation>> x=randn(1000,1000,50);

0.1938 s 0.0560 s3.5x faster!

Inlining Simple Functions

1.1942 s 0.3065 s

functions are worth inlining: - conv, cross, fft2, fliplr, flipud, ifft, ifft2, ifftn, ind2sub, ismember, linspace, logspace, mean, median, meshgrid, poly, polyval, repmat, roots, rot90, setdiff, setxor, sortrows, std, sub2ind, union, unique, var

y = medfilt1(x,5); 0.2082 s

Using the Right Type of Data“Do not use a cannon to kill a mosquito.”

double image: 0.5295 s

uint8 image: 0.1676 s

Confucius 

Matlab Organize its Arrays as Column-Major• Assign A to zero row-by-row or column-by-column>> n = 1e4;>> A = randn(n, n);

0.1041 s2.1740 s 21x faster!

Column-Major Memory Storage>> x = magic(3)x = 8 1 6 3 5 7 4 9 2

% Access one column>> y = x(:, 1);

% Access one row>> y = x(1, :);

Copy-on-Write (COW)>> n = 500;>> A = randn(n,n,n);

0.4794 s 0.0940 s

Clip values>> n = 2000;>> lowerBound = 0;>> upperBound = 1;>> A = randn(n,n);

0.0121 s0.1285 s 10x faster!

Moving Average Filter• Compute an N-sample moving average of x>> n = 1e7;>> N = 1000;>> x = randn(n,1);

3.2285 s 0.3847 s

Find the min/max of a matrix or N-d array >> n = 500;>> A = randn(n,n,n);

0.5465 s

0.1938 s

Acceleration using MEX (Matlab Executable)• Call your C, C++, or Fortran codes from the MATLAB

• Speed up specific subroutines

• See http://www.mathworks.com/help/matlab/matlab_external/introducing-mex-files.html

MATLAB Coder• MATLAB Coder™ generates standalone C and C++ code from

MATLAB® code

• See video examples in http://www.mathworks.com/products/matlab-coder/videos.html

• See http://www.mathworks.com/products/matlab-coder/

DoubleClass• http://documents.epfl.ch/users/l/le/leuteneg/www/MATLABToolbox/

DoubleClass.html

parfor for parallel processing• Requirements

• Task independent• Order independent

See http://www.mathworks.com/products/parallel-computing/

Parallel Processing in Matlab• MatlabMPI• multicore• pMatlab: Parallel Matlab Toolbox• Parallel Computing Toolbox (Mathworks)• Distributed Computing Server (Mathworks)• MATLAB plug-in for CUDA (CUDA is a library that used an nVidia board)

• Source: http://www-h.eng.cam.ac.uk/help/tpl/programs/Matlab/faster_scripts.html

Resources for your final projects• Awesome computer vision by Jia-Bin Huang

• A curated list of computer vision resources

• VLFeat• features extraction and matching, segmentation, clustering

• Piotr's Computer Vision Matlab Toolbox• Filters, channels, detectors, image/video manipulation

• OpenCV (MexOpenCV by Kota Yamaguchi)• General purpose computer vision library

Resources• Linear Algebra

• Linear Algebra Review and Reference• Linear algebra refresher course• Quick Review of Matrix and Real Linear Algebra

• MATLAB Tutorial• Resource collection• MATLAB F&Q

• Writing Fast MATLAB code• Techniques for Improving Performance by Mathwork• Writing Fast Matlab Code by Pascal Getreuer• Guidelines for writing clean and fast code in MATLAB by Nico Schlömer