Embed Size (px)
Citation preview
Lecture #2 - 8/31/2005 Slide 1 of 55
Introduction to SASVectors and Matrices
Lecture 2
August 31, 2005Multivariate Analysis
Overview
● Today’s Lecture
Introduction to SAS
Matrix Algebra
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 2 of 55
Today’s Lecture
■ Introduction to SAS.
■ Vectors and Matrices (Supplement 2A - augmented withSAS proc iml).
Lecture #2 - 8/31/2005 Slide 3 of 55
SAS■ To start SAS, on a Windows PC, go to Start...All Programs...The SAS
System...The SAS System for Windows V8.
Lecture #2 - 8/31/2005 Slide 4 of 55
Main Program Window■ The SAS program looks like this (some helpful commands are shown with
the arrows):
Overview
Introduction to SAS
● Main Program Window
● SAS Editor
● SAS Code
● First SAS Program
● Run
● SAS Data Libraries
● SAS Procedures
Matrix Algebra
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 5 of 55
SAS Editor■ To run SAS you must create a file of SAS code, which the
SAS processor reads and uses to run the program.
■ Simply type your SAS code into the Program Editor window.
■ For our example today, we will create (and save to) a newSAS code file, so to do that be sure to have your curserinside of the editor window and go to File...Save...
■ SAS code files usually end with the extension *.sas.
Overview
Introduction to SAS
● Main Program Window
● SAS Editor
● SAS Code
● First SAS Program
● Run
● SAS Data Libraries
● SAS Procedures
Matrix Algebra
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 6 of 55
SAS Code Basics■ For use in day-to-day statistical applications, SAS code
consists of two components:
◆ Data steps (where data input usually happens.
◆ Proc steps (where statistical analyses usually happen.
■ General exceptions to these rules exist.
■ All statements terminate with a semicolon (this is usuallywhere errors can occur).
■ Commented code can begin with:
◆ An asterisk (*) for single lines - terminated with asemicolon.
◆ /* for multiple lines, terminated with an ending */.
■ Enough talk...how about an example? Type the following intothe SAS Editor:
Lecture #2 - 8/31/2005 Slide 7 of 55
First SAS Programtitle ’First SAS Program’;data example1;input id gender $ age;cards;1 F 232 M 203 F 184 F .5 M 196 M 217 M 258 F 249 F 2010 M 22;
proc print;var id gender age;run;
Overview
Introduction to SAS
● Main Program Window
● SAS Editor
● SAS Code
● First SAS Program
● Run
● SAS Data Libraries
● SAS Procedures
Matrix Algebra
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 8 of 55
First SAS ProgramNotice the color scheme of the SAS Enhanced Editor (note: ifyou do not see color, do not panic, you may not have theEnhanced Editor installed).
■ Items in light blue are command words (like “input” or “var”.
■ Items in dark blue are procedural words (like “proc,” “data,” or“run”).
Overview
Introduction to SAS
● Main Program Window
● SAS Editor
● SAS Code
● First SAS Program
● Run
● SAS Data Libraries
● SAS Procedures
Matrix Algebra
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 9 of 55
Run the SAS Program■ To run the program you just entered, press the running man
icon at the center of the top of the SAS main program.
■ Once you press the “run” button, the log window will becomeactive, giving you information about the program as itexecutes.
■ In this window you will see errors (in red), or warnings (ingreen I think).
■ As multivariate progresses, you will become aware ofinstances where warnings will be present because ofproblems in your analysis.
■ Also now appearing is a new window called output...this iswhere the output of the procedure that was just run isdisplayed.
Overview
Introduction to SAS
● Main Program Window
● SAS Editor
● SAS Code
● First SAS Program
● Run
● SAS Data Libraries
● SAS Procedures
Matrix Algebra
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 10 of 55
SAS Data Libraries■ The data set you just entered is now part of a SAS data
library that can be referenced at any point during theremainder of the program.
■ The default SAS data library is called “work,” and can beaccessed by clicking through the explorer window on the lefthand side of the program.
■ Double click on Libraries...Work...Example1 and you will seethe data displayed in a grid.
■ To familiarize you with SAS, here are some handouts (alsoavailable on the BlackBoard Site)...
Lecture #2 - 8/31/2005 Slide 11 of 55
SAS Procedures■ The bulk of the statistical work done in SAS is through procedure statements.
■ Proc statements follow a flexible syntax that typically has the following:
proc -statement_name- data=-data_name- [options];var [included variables];[options];run;
■ The names and options are all found in the SAS manual, which is freely(shhh) available online at:http://www.id.unizh.ch/software/unix/statmath/sas/sasdoc/main.htm
Lecture #2 - 8/31/2005 Slide 12 of 55
SAS Procedures Example■ Using the Example1 data set, type the following into the editor:
proc sort data=example1;by gender;run;
proc univariate data=example1 plots;by gender;var age;run;
■ The univariate procedure produces univariate statistics, the manual entry canbe at found:
http://www.id.unizh.ch/software/unix/statmath/sas/sasdoc/proc/z0146802.htm
Overview
Introduction to SAS
Matrix Algebra
● Introduction
● Why?
● Definitions
● Matrix Computing
● Proc IML Basics
● Matrices
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 13 of 55
Matrix Introduction
■ Imagine you are interested in studying the culturaldifferences in student achievement in high school.
■ Your study gets IRB approval and you work hard to getparental approval.
■ You go into school and collect data using many differentinstruments.
■ You then input the data into your favorite stat package, likeSAS (or MS Excel).
■ How do you think the data is stored?
Overview
Introduction to SAS
Matrix Algebra
● Introduction
● Why?
● Definitions
● Matrix Computing
● Proc IML Basics
● Matrices
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 14 of 55
Why Learn Matrix Algebra?
■ Nearly all multivariate techniques are founded in matrixalgebra.
■ Often times when statistics break down, the cause of thefailure can be traced through matrix procedures.
■ If you are trying to apply a new method, most of the technicalstatistical literature uses matrix algebra assuming a basic toadvanced knowledge of matrices - you may need to readthese articles.
■ Have you seen:◆ (X′X)−1X′y?◆ ΛΦΛ
′ + Ψ?◆ F1F2F3LF−1
3 F−12 P(F−1
2 )′(F−13 )′L′F′
3F′
2F′
1 + U2?
■ Warning: using matrix algebra lingo is a great way to endconversations or break up parties.
Overview
Introduction to SAS
Matrix Algebra
● Introduction
● Why?
● Definitions
● Matrix Computing
● Proc IML Basics
● Matrices
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 15 of 55
Definitions
■ We begin this class with some general definitions (fromdictionary.com):
◆ Matrix:
1. A rectangular array of numeric or algebraic quantitiessubject to mathematical operations.
2. The substrate on or within which a fungus grows.
◆ Algebra:
1. A branch of mathematics in which symbols, usuallyletters of the alphabet, represent numbers or membersof a specified set and are used to represent quantitiesand to express general relationships that hold for allmembers of the set.
2. A set together with a pair of binary operations definedon the set. Usually, the set and the operations includean identity element, and the operations arecommutative or associative.
Overview
Introduction to SAS
Matrix Algebra
● Introduction
● Why?
● Definitions
● Matrix Computing
● Proc IML Basics
● Matrices
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 16 of 55
Proc IML
■ To help demonstrate the topics we will discuss today, I will beshowing examples in SAS proc iml.
■ The Interactive Matrix Language (IML) is a scientificcomputing package in SAS that typically used forcomplicated statistical routines.
■ Of course, other matrix programs exist - for many statisticalapplications MATLAB is very useful.
■ SPSS and SAS both have matrix computing capabilities, but(in my opinion) neither are as efficient, as user friendly, or asflexible as MATLAB.
◆ It is better to leave most of the statistical computing to thecomputer scientists.
Overview
Introduction to SAS
Matrix Algebra
● Introduction
● Why?
● Definitions
● Matrix Computing
● Proc IML Basics
● Matrices
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 17 of 55
Proc IML Basics
■ Proc IML is a proc step in SAS that runs without needing touse a preliminary data step.
■ To use IML, make sure the following are placed in a SAScode file.
proc iml;
reset print;
quit;
■ The “reset print;” line makes every result get printed in theoutput window.
■ The IML code will go between the “reset print;” and the “quit;”
Overview
Introduction to SAS
Matrix Algebra
● Introduction
● Why?
● Definitions
● Matrix Computing
● Proc IML Basics
● Matrices
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 18 of 55
Matrices
■ Away from the definition, a matrix is simply a rectangular wayof storing data.
■ Matrices can have unlimited dimensions, however for ourpurposes, all matrices will be in two dimensions:
◆ Rows
◆ Columns
■ Matrices are symbolized by boldface font in text, usuallywith capital letters.
A =
[
4 7 5
6 6 3
]
Overview
Introduction to SAS
Matrix Algebra
● Introduction
● Why?
● Definitions
● Matrix Computing
● Proc IML Basics
● Matrices
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 19 of 55
Vectors
■ A vector is a matrix where one dimension is equal to sizeone.
◆ Column vector: A column vector is a matrix of size r × 1.
◆ Row vector: A row vector is a matrix of size 1 × c.
■ Vectors allow for geometric representations of matrices.
■ The Pearson correlation coefficient is a function of the anglebetween vectors.
■ Much of the statistical theory underlying linear models(ANOVA-type) can be conceptualized by projections ofvectors (think of the dependent variable Y as a columnvector).
■ Vectors are typically symbolized by boldface font in text,usually with lowercase letters.
Overview
Introduction to SAS
Matrix Algebra
● Introduction
● Why?
● Definitions
● Matrix Computing
● Proc IML Basics
● Matrices
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 20 of 55
Scalars
■ A scalar is a matrix of size 1 × 1.
■ Scalars can be thought of as any single value.
■ The difficult concept to get used to is seeing a number as amatrix:
A =[
2.759]
Overview
Introduction to SAS
Matrix Algebra
● Introduction
● Why?
● Definitions
● Matrix Computing
● Proc IML Basics
● Matrices
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 21 of 55
Matrix Elements
■ A matrix is composed of a set of elements, each denoted it’srow and column position within the matrix.
■ For a matrix A of size r × c, each element is denoted by:
aij
◆ The first subscript is the index for the rows: i = 1, . . . , r.◆ The second subscript is the index for the columns:
j = 1, . . . , c.
A =
a11 a12 . . . a1c
a21 a22 . . . a1c
......
......
ar1 ar2 . . . arc
Overview
Introduction to SAS
Matrix Algebra
● Introduction
● Why?
● Definitions
● Matrix Computing
● Proc IML Basics
● Matrices
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 22 of 55
Transpose
■ The transpose of a matrix is simply the switching of theindices for rows and columns.
■ An element aij in the original matrix (in the ith row and jth
column) would be aji in the transposed matrix (in the jth rowand the ith column).
■ If the original matrix was of size i × j the transposed matrixwould be of size j × i.
A =
[
4 7 5
6 6 3
]
A′ =
4 6
7 6
5 3
Overview
Introduction to SAS
Matrix Algebra
● Introduction
● Why?
● Definitions
● Matrix Computing
● Proc IML Basics
● Matrices
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 23 of 55
Types of Matrices
■ Square Matrix: A matrix that as the same number of rowsand columns.
◆ Correlation and covariance matrices are examples ofsquare matrices.
■ Diagonal Matrix: A diagonal matrix is a square matrix withnon-zero elements down the diagonal and zero values forthe off-diagonal elements.
A =
2.759 0 0
0 1.643 0
0 0 0.879
■ Symmetric Matrix: A symmetric matrix is a square matrixwhere aij = aji for all elements in i and j.
◆ Correlation/covariance and distance matrices areexamples of symmetric matrices.
Overview
Introduction to SAS
Matrix Algebra
Algebra
● Algebra
● Addition
● Subtraction
● Multiplication
● Identity
● Zero
● “Division”
● Singular Matrices
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 24 of 55
Algebraic Operations
■ As mentioned in the definition at the beginning of class,algebra is simply a set of math that defines basic operations.
◆ Identity
◆ Zero
◆ Addition
◆ Subtraction
◆ Multiplication
◆ Division
■ Matrix algebra is simply the use of these operations withmatrices.
Overview
Introduction to SAS
Matrix Algebra
Algebra
● Algebra
● Addition
● Subtraction
● Multiplication
● Identity
● Zero
● “Division”
● Singular Matrices
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 25 of 55
Matrix Addition
■ Matrix addition is very much like scalar addition, the onlyconstraint is that the two matrices must be of the same size(same number of rows and columns).
■ The resulting matrix contains elements that are simply theresult of adding two scalars.
Overview
Introduction to SAS
Matrix Algebra
Algebra
● Algebra
● Addition
● Subtraction
● Multiplication
● Identity
● Zero
● “Division”
● Singular Matrices
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 26 of 55
Matrix Addition
A =
a11 a12
a21 a22
a31 a32
a41 a42
B =
b11 b12
b21 b22
b31 b32
b41 b42
A + B =
a11 + b11 a12 + b12
a21 + b21 a22 + b22
a31 + b31 a32 + b32
a41 + b41 a42 + b42
Overview
Introduction to SAS
Matrix Algebra
Algebra
● Algebra
● Addition
● Subtraction
● Multiplication
● Identity
● Zero
● “Division”
● Singular Matrices
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 27 of 55
IML Addition
proc iml;reset print;
A={10 15, 11 9, 1 -6};B={5 2, 1 0, 10 7};C=A+B;
quit;
Overview
Introduction to SAS
Matrix Algebra
Algebra
● Algebra
● Addition
● Subtraction
● Multiplication
● Identity
● Zero
● “Division”
● Singular Matrices
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 28 of 55
Matrix Subtraction
■ Matrix subtraction is identical to matrix addition, with theexception that all elements of the new matrix are thesubtracted elements of the previous matrices.
■ Again, the only constraint is that the two matrices must be ofthe same size (same number of rows and columns).
■ The resulting matrix contains elements that are simply theresult of subtracting two scalars.
Overview
Introduction to SAS
Matrix Algebra
Algebra
● Algebra
● Addition
● Subtraction
● Multiplication
● Identity
● Zero
● “Division”
● Singular Matrices
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 29 of 55
Matrix Subtraction
A =
a11 a12
a21 a22
a31 a32
a41 a42
B =
b11 b12
b21 b22
b31 b32
b41 b42
A − B =
a11 − b11 a12 − b12
a21 − b21 a22 − b22
a31 − b31 a32 − b32
a41 − b41 a42 − b42
Overview
Introduction to SAS
Matrix Algebra
Algebra
● Algebra
● Addition
● Subtraction
● Multiplication
● Identity
● Zero
● “Division”
● Singular Matrices
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 30 of 55
IML Subtraction
proc iml;reset print;
A={10 15, 11 9, 1 -6};B={5 2, 1 0, 10 7};C=A-B;
quit;
Overview
Introduction to SAS
Matrix Algebra
Algebra
● Algebra
● Addition
● Subtraction
● Multiplication
● Identity
● Zero
● “Division”
● Singular Matrices
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 31 of 55
Matrix Multiplication
■ Unlike matrix addition and subtraction, matrix multiplicationis much more complicated.
■ Matrix multiplication results in a new matrix that can be ofdiffering size from either of the two original matrices.
■ Matrix multiplication is defined only for matrices where thenumber of columns of the first matrix is equal to the numberof rows of the second matrix.
■ The resulting matrix as the same number of rows as the firstmatrix, and the same number of columns as the secondmatrix.
A B = C(r × c) (c × k) (r × k)
Overview
Introduction to SAS
Matrix Algebra
Algebra
● Algebra
● Addition
● Subtraction
● Multiplication
● Identity
● Zero
● “Division”
● Singular Matrices
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 32 of 55
Matrix Multiplication
A =
a11 a12
a21 a22
a31 a32
a41 a42
B =
[
b11 b12 b13
b21 b22 b23
]
AB =
a11b11 + a12b21 a11b12 + a12b22 a11b13 + a12b23
a21b11 + a22b21 a21b12 + a22b22 a21b13 + a22b23
a31b11 + a32b21 a31b12 + a32b22 a31b13 + a32b23
a41b11 + a42b21 a41b12 + a42b22 a41b13 + a42b23
Overview
Introduction to SAS
Matrix Algebra
Algebra
● Algebra
● Addition
● Subtraction
● Multiplication
● Identity
● Zero
● “Division”
● Singular Matrices
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 32 of 55
Matrix Multiplication
A =
a11 a12
a21 a22
a31 a32
a41 a42
B =
[
b11 b12 b13
b21 b22 b23
]
AB =
a11b11 + a12b21 a11b12 + a12b22 a11b13 + a12b23
a21b11 + a22b21 a21b12 + a22b22 a21b13 + a22b23
a31b11 + a32b21 a31b12 + a32b22 a31b13 + a32b23
a41b11 + a42b21 a41b12 + a42b22 a41b13 + a42b23
Overview
Introduction to SAS
Matrix Algebra
Algebra
● Algebra
● Addition
● Subtraction
● Multiplication
● Identity
● Zero
● “Division”
● Singular Matrices
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 32 of 55
Matrix Multiplication
A =
a11 a12
a21 a22
a31 a32
a41 a42
B =
[
b11 b12 b13
b21 b22 b23
]
AB =
a11b11 + a12b21 a11b12 + a12b22 a11b13 + a12b23
a21b11 + a22b21 a21b12 + a22b22 a21b13 + a22b23
a31b11 + a32b21 a31b12 + a32b22 a31b13 + a32b23
a41b11 + a42b21 a41b12 + a42b22 a41b13 + a42b23
Overview
Introduction to SAS
Matrix Algebra
Algebra
● Algebra
● Addition
● Subtraction
● Multiplication
● Identity
● Zero
● “Division”
● Singular Matrices
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 33 of 55
IML Multiplication
proc iml;reset print;
A={10 15, 11 9, 1 -6};B={5 2, 1 0, 10 7};C=A*T(B);D=T(B)*(A);
quit;
Overview
Introduction to SAS
Matrix Algebra
Algebra
● Algebra
● Addition
● Subtraction
● Multiplication
● Identity
● Zero
● “Division”
● Singular Matrices
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 34 of 55
Multiplication and Summation
■ Because of the additive nature induced by matrixmultiplication, many statistical formulas that use:
∑
can be expressed by matrix notation.
■ For instance, consider a single variable Xi, with i = 1, . . . , Nobservations.
■ Putting the set of observations into the column vector X, ofsize N × 1, we can show that:
N∑
i=1
X2 = X′ X
Overview
Introduction to SAS
Matrix Algebra
Algebra
● Algebra
● Addition
● Subtraction
● Multiplication
● Identity
● Zero
● “Division”
● Singular Matrices
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 35 of 55
Matrix Multiplication by Scalar
■ Recall that a scalar is simply a matrix of size (1 × 1).■ Matrix multiplication by a scalar causes all elements of the
matrix to be multiplied by the scalar.■ The resulting matrix has all elements multiplied by the scalar.
A =
a11 a12
a21 a22
a31 a32
a41 a42
s × A =
s × a11 s × a12
s × a21 s × a22
s × a31 s × a32
s × a41 s × a42
Overview
Introduction to SAS
Matrix Algebra
Algebra
● Algebra
● Addition
● Subtraction
● Multiplication
● Identity
● Zero
● “Division”
● Singular Matrices
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 36 of 55
Identity Matrix
■ The identity matrix is defined as a matrix that whenmultiplied with another matrix produces that original matrix:
A I = A
I A = A
■ The identity matrix is simply a square matrix that has alloff-diagonal elements equal to zero, and all diagonalelements equal to one.
I(3×3) =
1 0 0
0 1 0
0 0 1
Overview
Introduction to SAS
Matrix Algebra
Algebra
● Algebra
● Addition
● Subtraction
● Multiplication
● Identity
● Zero
● “Division”
● Singular Matrices
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 37 of 55
Zero Matrix
■ The zero matrix is defined as a matrix that when multipliedwith another matrix produces the matrix:
A 0 = 0
0 A = 0
■ The zero matrix is simply a square matrix that has allelements equal to zero.
0(3×3) =
0 0 0
0 0 0
0 0 0
Overview
Introduction to SAS
Matrix Algebra
Algebra
● Algebra
● Addition
● Subtraction
● Multiplication
● Identity
● Zero
● “Division”
● Singular Matrices
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 38 of 55
Matrix “Division”: The Inverse
■ Recall from basic math that:
a
b=
1
ba = b−1a
■ And that:a
a= 1
■ Matrix inverses are just like division in basic math.
Overview
Introduction to SAS
Matrix Algebra
Algebra
● Algebra
● Addition
● Subtraction
● Multiplication
● Identity
● Zero
● “Division”
● Singular Matrices
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 39 of 55
The Inverse
■ For a square matrix, an inverse matrix is simply the matrixthat when pre-multiplied with another matrix produces theidentity matrix:
A−1A = I
■ Matrix inverse calculation is complicated and unnecessarysince computers are much more efficient at finding inversesof matrices.
■ One point of emphasis: just like in regular division, divisionby zero is undefined.
■ By analogy - not all matrices can be inverted.
Overview
Introduction to SAS
Matrix Algebra
Algebra
● Algebra
● Addition
● Subtraction
● Multiplication
● Identity
● Zero
● “Division”
● Singular Matrices
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 40 of 55
Singular Matrices
■ A matrix that cannot be inverted is called a singular matrix.
■ In statistics, common causes of singular matrices are foundby linear dependence among the rows or columns of asquare matrix.
■ Linear dependence can be cause by combinations ofvariables, or by variables with extreme correlations (eithernear 1.00 or -1.00).
Overview
Introduction to SAS
Matrix Algebra
Algebra
Vector Definitions
● Linear Combinations
● Linear Dependencies
● Vector Length
● Inner Product
● Angle Between Vectors
● Vector Projections
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 41 of 55
Linear Combinations
■ Vectors can be combined by adding multiples:
y = a1x1 + a2x2 + . . . + akxk
■ The resulting vector, y, is called a linear combination.
■ All for k vectors, the set of all possible linear combinations iscalled their span.
Overview
Introduction to SAS
Matrix Algebra
Algebra
Vector Definitions
● Linear Combinations
● Linear Dependencies
● Vector Length
● Inner Product
● Angle Between Vectors
● Vector Projections
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 42 of 55
Linear Dependencies
■ A set of vectors are said to be linearly dependent ifa1, a2, . . . , ak exist, and:
◆ a1x1 + a2x2 + . . . + akxk = 0.
◆ a1, a2, . . . , ak are not all zero.
■ Such linear dependencies occur when a linear combinationis added to the vector set.
■ Matrices comprised of a set of linearly dependent vectorsare singular.
■ A set of linearly independent vectors forms what is called abasis for the vector space.
■ Any vector in the vector space can then be expressed as alinear combination of the basis vectors.
Overview
Introduction to SAS
Matrix Algebra
Algebra
Vector Definitions
● Linear Combinations
● Linear Dependencies
● Vector Length
● Inner Product
● Angle Between Vectors
● Vector Projections
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 43 of 55
Vector Length
■ The length of a vector emanating from the origin is given bythe Pythagorean formula:
Lx =√
x21 + x2
2 + . . . + x2k
Overview
Introduction to SAS
Matrix Algebra
Algebra
Vector Definitions
● Linear Combinations
● Linear Dependencies
● Vector Length
● Inner Product
● Angle Between Vectors
● Vector Projections
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 44 of 55
Inner Product
■ The inner (or dot) product of two vectors x and y is the sumof element-by-element multiplication:
x′y = x1y1 + x2y2 + . . . + xkyk
Overview
Introduction to SAS
Matrix Algebra
Algebra
Vector Definitions
● Linear Combinations
● Linear Dependencies
● Vector Length
● Inner Product
● Angle Between Vectors
● Vector Projections
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 45 of 55
Vector Angle
■ The angle formed between two vectors x and y is
cos(θ) =x′y√
x′x√
y′y
■ If x′y = 0, vectors x and y are perpendicular, as noted byx⊥y
■ All basis vectors are perpendicular.
Overview
Introduction to SAS
Matrix Algebra
Algebra
Vector Definitions
● Linear Combinations
● Linear Dependencies
● Vector Length
● Inner Product
● Angle Between Vectors
● Vector Projections
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 46 of 55
Vector Projections
■ The projection of a vector x onto a vector y is given by:
x′yL2
yy
■ Through such projections, a set of linear independentvectors can be created from any set of vectors.
■ One process used to create such vectors is through theGram-Schmidt Process.
Overview
Introduction to SAS
Matrix Algebra
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 47 of 55
Matrix Properties
The following are some algebraic properties of matrices:
■ (A + B) + C = A + (B + C) - Associative
■ A + B = B + A - Commutative
■ c(A + B) = cA + cB - Distributive
■ (c + d)A = cA + dA
■ (A + B)′ = A′ + B′
■ (cd)A = c(dA)
■ (cA)′ = cA′
Overview
Introduction to SAS
Matrix Algebra
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
Lecture #2 - 8/31/2005 Slide 48 of 55
Matrix Properties
The following are more algebraic properties of matrices:
■ c(AB) = (cA)B
■ A(BC) = (AB)C
■ A(B + C) = AB + AC
■ (B + C)A = BA + CA
■ (AB)′ = B′A′
■ For xj such that Ax j is defined:
◆
n∑
j=1
Ax j = An
∑
j=1
xj
◆
n∑
j=1
(Ax j)(Ax j)′ = A
n∑
j=1
xjx′
j
A′
Overview
Introduction to SAS
Matrix Algebra
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
● Advanced Topics
● Determinants
● Orthogonality
● Eigenspaces
● Decompositions
Wrapping Up
Lecture #2 - 8/31/2005 Slide 49 of 55
Advanced Matrix Functions/Operations
■ We end our matrix discussion with some advanced topics.
■ All of these topics are related to multivariate analyses.
■ None of these will seem too straight forward today, but nextweek we will use some of these results to demonstrateproperties of sample statistics.
Overview
Introduction to SAS
Matrix Algebra
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
● Advanced Topics
● Determinants
● Orthogonality
● Eigenspaces
● Decompositions
Wrapping Up
Lecture #2 - 8/31/2005 Slide 50 of 55
Matrix Determinants
■ A square matrix can be characterized by a scalar valuecalled a determinant.
det A = |A|■ Much like the matrix inverse, calculation of the determinant is
very complicated and tedious, and is best left to computers.
■ What can be learned from determinants is if a matrix issingular.
■ Matrices with determinants that are greater than zero aresaid to be “positive definite,” a byproduct of which is that apositive matrix is non-singular.
Overview
Introduction to SAS
Matrix Algebra
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
● Advanced Topics
● Determinants
● Orthogonality
● Eigenspaces
● Decompositions
Wrapping Up
Lecture #2 - 8/31/2005 Slide 51 of 55
Matrix Orthogonality
■ A square matrix (A) is said to be orthogonal if:
AA ′ = A′A = I
■ Orthogonal matrices are characterized by two properties:
1. The product of all row vector multiples is the zero matrix(perpendicular vectors).
2. For each row vector, the sum of all elements is one.
Overview
Introduction to SAS
Matrix Algebra
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
● Advanced Topics
● Determinants
● Orthogonality
● Eigenspaces
● Decompositions
Wrapping Up
Lecture #2 - 8/31/2005 Slide 52 of 55
Eigenvalues and Eigenvectors
■ A square matrix can be decomposed into a set ofeigenvalues and eigenvectors.
Ax = λx
■ From a statistical standpoint:◆ Principal components are comprised of linear combination
of a set of variables weighed by the eigenvectors.
◆ The eigenvalues represent the proportion of varianceaccounted for by specific principal components.
◆ Each principal component is orthogonal to the next,producing a set of uncorrelated variables that may beused for regression purposes.
Overview
Introduction to SAS
Matrix Algebra
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
● Advanced Topics
● Determinants
● Orthogonality
● Eigenspaces
● Decompositions
Wrapping Up
Lecture #2 - 8/31/2005 Slide 53 of 55
Spectral Decompositions
■ Imagine that a matrix A is of size k × k.
■ A then has:
◆ k eigenvalues: λi, i = 1, . . . , k.
◆ k eigenvectors: ei, i = 1, . . . , k (each of size k × 1.
■ A can be expressed by:
A =
k∑
i=1
λieie′
i
■ This expression is called the Spectral Decomposition, whereA is decomposed into k parts.
■ One can find A−1 by taking 1λ
in the spectral decomposition.
Overview
Introduction to SAS
Matrix Algebra
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
● Final Thought
● Next Class
Lecture #2 - 8/31/2005 Slide 54 of 55
Final Thought
■ The SAS learning curve issmall, but once you havethe basics, all you need isthe manual and you can doalmost anything.
■ Proc IML is useful formatrices, but probably notso useful to you.
■ Matrix algebra makes the technical things in life easier.
■ The applications of matrices will be demonstrated throughoutthe rest of this course.
Overview
Introduction to SAS
Matrix Algebra
Algebra
Vector Definitions
Matrix Properties
Advanced Topics
Wrapping Up
● Final Thought
● Next Class
Lecture #2 - 8/31/2005 Slide 55 of 55
Next Time
■ Applications of matrix algebra in statistics (Chapter 2 sectionsix and on).
■ Geometric implications of multivariate descriptive statistics(more applications).
