Prelude A pattern of activation in a NN is a vector A set of connection weights between units is a matrix Vectors and matrices have well-understood mathematical

Prelude• A pattern of activation in a NN is a vector• A set of connection weights between units is

a matrix• Vectors and matrices have well-understood

mathematical and geometric properties• Very useful for understanding the properties

of NNs

Operations on Vectors and Matrices

Outline1) The Players: Scalars, Vectors and Matrices

2) Vectors, matrices and neural nets

3) Geometric Analysis of Vectors

4) Multiplying Vectors by Scalars

5) Multiplying Vectors by Vectors

a) The inner product (produces a scalar)

b) The outer product (produces a matrix)

6) Multiplying Vectors by Matrices

7) Multiplying Matrices by Matrices

Scalars, Vectors and Matrices1) Scalar: A single number (integer or real)

2) Vector: An ordered list of scalars

[ 1 2 3 4 5 ] [ 0.4 1.2 0.07 8.4 12.3 ] [ 12 10 ] [ 2 ]



[ 1 2 3 4 5 ] [ 0.4 1.2 0.07 8.4 12.3 ] [ 12 10 ] [ 2 ]

[ 12 10 ] ≠ [ 10 12 ]



[ 1 2 3 4 5 ] [ 0.4 1.2 0.07 8.4 12.3 ] [ 12 10 ] [ 2 ]

Row vectors



[ 1 2 3 4 5 ] [ 0.4 1.2 0.07 8.4 12.3 ] [ 12 10 ] [ 2 ]

Row vectors

Column Vectors12345

1.5

0.3

6.2

12.0

17.1



3) Matrix: An ordered list of vectors:

1 2 6 1 7 8

2 5 9 0 0 3

3 1 5 7 6 3

2 7 9 3 3 1




1 2 6 1 7 8

2 5 9 0 0 3

3 1 5 7 6 3

2 7 9 3 3 1

Row vectors




1 2 6 1 7 8

2 5 9 0 0 3

3 1 5 7 6 3

2 7 9 3 3 1

Column vectors




1 2 6 1 7 8

2 5 9 0 0 3

3 1 5 7 6 3

2 7 9 3 3 1

Matrices are indexed (row, column)

M =




1 2 6 1 7 8

2 5 9 0 0 3

3 1 5 7 6 3

2 7 9 3 3 1


M =M(1,3) = 6 (row 1, column 3)




1 2 6 1 7 8

2 5 9 0 0 3

3 1 5 7 6 3

2 7 9 3 3 1


M =M(1,3) = 6 (row 1, column 3)

M(3,1) = 3 (row 3, column 1)

Variable Naming Conventions1) Scalars: Lowercase, italics

x, y, z…

2) Vectors: Lowercase, bold

u, v, w…

• Matrices: Uppercase, bold

M, N, O …

• Constants: Greek

, , , , …

Transposing VectorsIf u is a row vector…

u = [ 1 2 3 4 5 ]

…then u’ (“u-transpose”) is a column vector

12345

… and vice-versa.

u’ =


u = [ 1 2 3 4 5 ]


12345

… and vice-versa.

u’ =Why in the world

would I care??

You


u = [ 1 2 3 4 5 ]


12345

… and vice-versa.

u’ = Answer: It’ll matter when we come to vector multiplication.


u = [ 1 2 3 4 5 ]


12345

… and vice-versa.

u’ = OK.

Vectors, Matrices & Neural Nets


j1 j2 j3 Input units, j


i1 i2


Output units, i


i1 i2


Output units, i

Connection weights, wij

w11

w12 w13w21w22

w23


i1 i2

0.2 0.9 0.5 Input units, j

Output units, i


w11

w12 w13w21w22

w23

The activations of the input nodes can be represented as a 3-dimensional vector:

j = [ 0.2 0.9 0.5 ]


1.0 0.0


Output units, i


w11

w12 w13w21w22

w23

The activations of the output nodes can be represented as a 2-dimensional vector:

i = [ 1.0 0.0 ]


i1 i2


Output units, i


w11

w12 w13w21w22

w23

The weights leading into any output node can be represented as a 3-dimensional vector:

w1j = [ 0.1 1.0 0.2 ]

0.1

1.0 0.2


i1 i2


Output units, i


w11

w12 w13w21w22

w23

The complete set of weights can be represented as a 3 (row) X 2 (column) matrix:

0.1

1.0 0.21.0 0.1

-0.9

W =0.1 1.0 0.2 1.0 0.1 -0.9


i1 i2


Output units, i


w11

w12 w13w21w22

w23

The complete set of weights can be represented as a 2 (row) X 3 (column) matrix:

0.1

1.0 0.21.0 0.1

-0.9

W =

Why in the world would I care??

0.1 1.0 0.2 1.0 0.1 -0.9


W

Why in the world would I care??

1. Because the mathematics of vectors and matrices is well-understood.

2. Because vectors have a very useful geometric interpretation.

3. Because Matlab “thinks” in vectors and matrices.

4. Because you are going to have to learn to think in Matlab.


OK.

1. Because the mathematics of vectors and matrices is well-understood.

2. Because vectors have a very useful geometric interpretation.

3. Because Matlab “thinks” in vectors and matrices.

4. Because you are going to have to learn to think in Matlab.

Geometric Analysis of VectorsDimensionality: The number of numbers in a vector

Geometric Analysis of Vectors

Vector as a point Vector as arrow

x1

x2

x3

.3

.2x1

x2

x3

.5

.3

.5

.2

Dimensionality: The number of numbers in a vector


Vector as a point Vector as arrow

x1

x2

x3

.3

.2x1

x2

x3

.5

.3

.5

.2

Dimensionality: The number of numbers in a vector


Implications for neural networks

• Auto-associative nets• State of activation at time t is a vector (a point in a space)

• As activations change, vector moves through that space

• Will prove invaluable in understanding Hopfield nets

• Layered nets (“perceptrons”)• Input vectors activate output vectors

• Points in input space map to points in output space

• Will prove invaluable in understanding perceptrons and back-propagation learning

Multiplying a Vector by a Scalar

[ 5 4 ] * 2 =

5

4

Multiplying a Vector by a Scalar

[ 5 4 ] * 2 = [ 10 8 ]

Lengthens the vector but does not change its orientation

5

4

10

8

Adding a Vector to a Scalar

[ 5 4 ] + 2 =

5

4

Adding a Vector to a Scalar

[ 5 4 ] + 2 = NAN

Is Illegal.

5

4

Adding a Vector to a Vector

[ 5 4 ] + [ 3 6 ] =

5

4

3

6

Adding a Vector to a Vector

[ 5 4 ] + [ 3 6 ] = [ 8 10 ]

Forms a parallelogram.

5

4

3

6

8

10

Multiplying a Vector by a Vector 1:The Inner Product (aka “Dot Product”)

If u and v are both row vectors of the same dimensionality…

u = [ 1 2 3 ]

v = [ 4 5 6 ]



u = [ 1 2 3 ]

v = [ 4 5 6 ]

… then the product

u · v =



u = [ 1 2 3 ]

v = [ 4 5 6 ]


u · v = NAN

Is undefined.



u = [ 1 2 3 ]

v = [ 4 5 6 ]


u · v = NAN

Is undefined.

Huh??Why??

That’s BS!


I told you you’d eventually care about transposing vectors…

?


• The Mantra: “Rows by Columns”• Multiply rows (or row vectors) by columns (or

column vectors)



column vectors)

u = [ 1 2 3 ]

v = [ 4 5 6 ]



column vectors)

u = [ 1 2 3 ]

v = [ 4 5 6 ] v’ =

4

5

6



column vectors)

u = [ 1 2 3 ]

v’ =

4

5

6

u · v’ = 32



column vectors)

u = [ 1 2 3 ]

v’

4

5

6

u · v’ = 32

Imagine rotating your row vector into a (pseudo) column vector…

1

2

3



column vectors)

u = [ 1 2 3 ]

v’

4

5

6

u · v’ = 32

Now multiply corresponding elements and add up the products…

1

2

3

4



column vectors)

u = [ 1 2 3 ]

v’

4

5

6

u · v’ = 32


1

2

3

4

10



column vectors)

u = [ 1 2 3 ]

v’

4

5

6

u · v’ = 32


1

2

3

4

10

18



column vectors)

u = [ 1 2 3 ]

v’

4

5

6

u · v’ = 32


1

2

3

4

10

1832

4

5

6

4

10

1832


• Inner product is commutative as long as you transpose correctly

u = [ 1 2 3 ]

v’

u · v’ = 32

v · u’ = 32

u’

1

2

3

v = [ 4 5 6 ]

v

4

5

6

4

10

1832

The Inner (“Dot”) Product

• In scalar notation…v’

4

5

6

1

2

3

4

10

1832

u

i

d

iivuv'u

1


• In scalar notation…

• Remind you of…

… the net input to a unit

i

d

iivuv'u

1

j

d

jiji awn

1


• In scalar notation…

• Remind you of…

j

d

jijij awn

1

… the net input to a unit

• In vector notation…

awn

i

d

iivuv'u

1

What Does the Dot Product “Mean”?


• Consider u u’


• Consider u u’

u = [ 3, 4 ]

3

4


• Consider u u’

u = [ 3, 4 ]

u’

3

4

3

4

9

16

25

u

3

4


• Consider u u’

u = [ 3, 4 ]

u’

3

4

3

4

9

16

25

u

3

4

u uu'

u ui2

5


• Consider u u’

u = [ 3, 4 ]

u’

3

4

3

4

9

16

25

u

3

4

u uu'

u ui2

5True for vectors of any dimensionality


• So:

u uu'


• What about u v where u v?


cos(uv ) uvu v

Well…



cos(uv ) uvu v

Well…


… and cos(uv) is a length-invariant measure of the similarity of u and v


cos(uv ) uvu v


cos(uv) is a length-invariant measure of the similarity of u and v

U = [ 1, 0 ]

V’ = [ 1, 1 ]


cos(uv ) uvu v



U = [ 1, 0 ]

V’ = [ 1, 1 ]

uv = 45º; cos(uv) = .707

U V = (1 * 1) + (1 * 0) = 1


cos(uv ) uvu v



U = [ 1, 0 ]

V’ = [ 1, 1 ] U V = (1 * 1) + (1 * 0) = 1

cos(uv ) 1

u v

uv = 45º; cos(uv) = .707


cos(uv ) uvu v



U = [ 1, 0 ]

V’ = [ 1, 1 ]||u|| = sqrt(1) = 1

||v|| = sqrt(2) = 1.414

cos(uv ) 1

u v

uv = 45º; cos(uv) = .707


cos(uv ) uvu v



U = [ 1, 0 ]

V’ = [ 1, 1 ]||u|| = sqrt(1) = 1

||v|| = sqrt(2) = 1.414

cos(uv ) 1

1*1.414.707

uv = 45º; cos(uv) = .707


cos(uv ) uvu v



U = [ 1, 0 ]

V’ = [ 0, 1 ]

uv = 90º; cos(uv) = 0


cos(uv ) uvu v



U = [ 1, 0 ]

V’ = [ 0, -1 ]

uv = 270º; cos(uv) = 0


cos(uv ) uvu v



U = [ 1, 0 ]V’ = [ -1,0 ]

uv = 180º; cos(uv) = -1


cos(uv ) uvu v



U = [ 1, 0 ]

V’ = [ 2.2,0 ]

uv = 0º; cos(uv) = 1


cos(uv ) uvu v



In general…

cos(uv) -1…1

True regardless of dimensionality


cos(uv ) uvu v



To see why, consider the cosine expressed in scalar notation…

cos(uv ) uiv i

ui2 v i

2


cos(uv ) uvu v



… and compare it to the equation for the correlation coefficient…

cos(uv ) uiv i

ui2 v i

2

r(u,v) (ui u )(v i v )

(ui u )2 (v i v )2


cos(uv ) uvu v




cos(uv ) uiv i

ui2 v i

2


(ui u )2 (v i v )2

if u and v have means of zero, then cos(uv) = r(u,v)


cos(uv ) uvu v




cos(uv ) uiv i

ui2 v i

2


(ui u )2 (v i v )2

if u and v have means of zero, then cos(uv) = r(u,v)

The cosine is a special case of the correlation coefficient!


cos(uv ) uvu v



… and let’s compare the cosine to the dot product…


cos(uv ) uvu v




If u and v have lengths of 1, then the dot product is equal to the cosine.


cos(uv ) uvu v




If u and v have lengths of 1, then the dot product is equal to the cosine.

The dot product is a special case of the cosine, which is a special case of the correlation coefficient, which is a measure of vector similarity!


aw ijj

iji awn • The most common input rule is a dot product between unit i’s

vector of weights and the activation vector on the other end

• Such a unit is computing the (biased) similarity between what it expects (wi) and what it’s getting (a).

• It’s activation is a positive function of this similarity


• The most common input rule is a dot product between unit i’s vector of weights and the activation vector on the other end



ai

ni

asymptotic

aw ijj

iji awn





ai

ni

Step (BTU)

aw ijj

iji awn





ai

ni

logistic

aw ijj

iji awn

Multiplying a Vector by a Vector 2:The Outer Product

The two vectors need not have the same dimensionality.

Same Mantra: Rows by Columns.

This time, multiply a column vector by a row vector:





u’ =1

2v = [ 4 5 6 ]

M = u’ * v





1

2v = [ 4 5 6 ]

M = u’ * v

M =u’ =





1

2v = [ 4 5 6 ]

M = u’ * v

M =

Row 1

u’ =





1

2v = [ 4 5 6 ]

M = u’ * v

M =

Row 1 times column 1

u’ =





1

2v = [ 4 5 6 ]

M = u’ * v

M =

Row 1 times column 1 goes into row 1, column 1

4 u’ =





1

2v = [ 4 5 6 ]

M = u’ * v

M =


4 5 u’ =





1

2v = [ 4 5 6 ]

M = u’ * v

M =


4 5 6 u’ =





u =1

2v = [ 4 5 6 ]

M = u’ * v

M =


4 5 6 8





1

2v = [ 4 5 6 ]

M = u’ * v

M =


4 5 6 8 10

u’ =





1

2v = [ 4 5 6 ]

M = u’ * v

M =


4 5 6 8 10 12

u’ =

u’ =





12

v = [ 4 5 6 ]

M = u’ * v

= M4 5 6 8 10 12

A better way to visualize it…

12


Outer product is not exactly commutative…

u’ =

v = [ 4 5 6 ]

M = u’ * v

= M4 5 6 8 10 12

M = v’ * u

u = [ 1 2 ]

456

v’ =4 85 106 12

Multiplying a Vector by a Matrix• Same Mantra: Rows by Columns

Rows by Columns

A row vector:

[ .2 .6 .3 .7 .9 .4 .3 ]

Rows by Columns

A row vector:

[ .2 .6 .3 .7 .9 .4 .3 ]

A matrix:

.3 .4 .8 .1 .2 .3

.5 .2 0 .1 .5 .2

.1 .1 .9 .2 .5 .3

.2 .4 .1 .7 .8 .5

.9 .9 .2 .5 .3 .5

.4 .1 .2 .7 .8 .2

.1 .2 .2 .5 .7 .2

Rows by Columns

A row vector:

[ .2 .6 .3 .7 .9 .4 .3 ]

A matrix:

.3 .4 .8 .1 .2 .3

.5 .2 0 .1 .5 .2

.1 .1 .9 .2 .5 .3

.2 .4 .1 .7 .8 .5

.9 .9 .2 .5 .3 .5

.4 .1 .2 .7 .8 .2

.1 .2 .2 .5 .7 .2

Multiply rows

Rows by Columns

A row vector:

[ .2 .6 .3 .7 .9 .4 .3 ]

A matrix:

.3 .4 .8 .1 .2 .3

.5 .2 0 .1 .5 .2

.1 .1 .9 .2 .5 .3

.2 .4 .1 .7 .8 .5

.9 .9 .2 .5 .3 .5

.4 .1 .2 .7 .8 .2

.1 .2 .2 .5 .7 .2

Multiply rows by columns

Each such multiplication is a simple dot product

A row vector:

[ .2 .6 .3 .7 .9 .4 .3 ]

A matrix:

.3 .4 .8 .1 .2 .3

.5 .2 0 .1 .5 .2

.1 .1 .9 .2 .5 .3

.2 .4 .1 .7 .8 .5

.9 .9 .2 .5 .3 .5

.4 .1 .2 .7 .8 .2

.1 .2 .2 .5 .7 .2


A row vector:

[ .2 .6 .3 .7 .9 .4 .3 ]

A matrix:

.3 .4 .8 .1 .2 .3

.5 .2 0 .1 .5 .2

.1 .1 .9 .2 .5 .3

.2 .4 .1 .7 .8 .5

.9 .9 .2 .5 .3 .5

.4 .1 .2 .7 .8 .2

.1 .2 .2 .5 .7 .2

Make a proxy column vector…


A row vector:

[ .2 .6 .3 .7 .9 .4 .3 ]

A matrix:

.3 .4 .8 .1 .2 .3

.5 .2 0 .1 .5 .2

.1 .1 .9 .2 .5 .3

.2 .4 .1 .7 .8 .5

.9 .9 .2 .5 .3 .5

.4 .1 .2 .7 .8 .2

.1 .2 .2 .5 .7 .2

.2

.6

.3

.7

.9

.4

.3


A row vector:

[ .2 .6 .3 .7 .9 .4 .3 ]

A matrix:

.3 .4 .8 .1 .2 .3

.5 .2 0 .1 .5 .2

.1 .1 .9 .2 .5 .3

.2 .4 .1 .7 .8 .5

.9 .9 .2 .5 .3 .5

.4 .1 .2 .7 .8 .2

.1 .2 .2 .5 .7 .2

.2

.6

.3

.7

.9

.4

.3

Now compute the dot product of the (proxy) row vector with each column of the matrix…

[ 1.5 ]


A row vector:

[ .2 .6 .3 .7 .9 .4 .3 ]

A matrix:

.3 .4 .8 .1 .2 .3

.5 .2 0 .1 .5 .2

.1 .1 .9 .2 .5 .3

.2 .4 .1 .7 .8 .5

.9 .9 .2 .5 .3 .5

.4 .1 .2 .7 .8 .2

.1 .2 .2 .5 .7 .2

.2

.6

.3

.7

.9

.4

.3

[ 1.5 1.4 ]


A row vector:

[ .2 .6 .3 .7 .9 .4 .3 ]

A matrix:

.3 .4 .8 .1 .2 .3

.5 .2 0 .1 .5 .2

.1 .1 .9 .2 .5 .3

.2 .4 .1 .7 .8 .5

.9 .9 .2 .5 .3 .5

.4 .1 .2 .7 .8 .2

.1 .2 .2 .5 .7 .2

.2

.6

.3

.7

.9

.4

.3

[ 1.5 1.4 0.8 ]


A row vector:

[ .2 .6 .3 .7 .9 .4 .3 ]

A matrix:

.3 .4 .8 .1 .2 .3

.5 .2 0 .1 .5 .2

.1 .1 .9 .2 .5 .3

.2 .4 .1 .7 .8 .5

.9 .9 .2 .5 .3 .5

.4 .1 .2 .7 .8 .2

.1 .2 .2 .5 .7 .2

.2

.6

.3

.7

.9

.4

.3

[ 1.5 1.4 0.8 1.5 ]


A row vector:

[ .2 .6 .3 .7 .9 .4 .3 ]

A matrix:

.3 .4 .8 .1 .2 .3

.5 .2 0 .1 .5 .2

.1 .1 .9 .2 .5 .3

.2 .4 .1 .7 .8 .5

.9 .9 .2 .5 .3 .5

.4 .1 .2 .7 .8 .2

.1 .2 .2 .5 .7 .2

.2

.6

.3

.7

.9

.4

.3

[ 1.5 1.4 0.8 1.5 1.9 ]


A row vector:

[ .2 .6 .3 .7 .9 .4 .3 ]

A matrix:

.3 .4 .8 .1 .2 .3

.5 .2 0 .1 .5 .2

.1 .1 .9 .2 .5 .3

.2 .4 .1 .7 .8 .5

.9 .9 .2 .5 .3 .5

.4 .1 .2 .7 .8 .2

.1 .2 .2 .5 .7 .2

.2

.6

.3

.7

.9

.4

.3

[ 1.5 1.4 0.8 1.5 1.9 1.2 ]

A row vector:

[ .2 .6 .3 .7 .9 .4 .3 ]

A matrix:

.3 .4 .8 .1 .2 .3

.5 .2 0 .1 .5 .2

.1 .1 .9 .2 .5 .3

.2 .4 .1 .7 .8 .5

.9 .9 .2 .5 .3 .5

.4 .1 .2 .7 .8 .2

.1 .2 .2 .5 .7 .2The result is a row vector with as many columns (dimensions) as the matrix (not the vector)

[ 1.5 1.4 0.8 1.5 1.9 1.2 ]

A row vector:

[ .2 .6 .3 .7 .9 .4 .3 ]

A matrix:

.3 .4 .8 .1 .2 .3

.5 .2 0 .1 .5 .2

.1 .1 .9 .2 .5 .3

.2 .4 .1 .7 .8 .5

.9 .9 .2 .5 .3 .5

.4 .1 .2 .7 .8 .2

.1 .2 .2 .5 .7 .2

[ 1.5 1.4 0.8 1.5 1.9 1.2 ]

7-dimensional vector

A row vector:

[ .2 .6 .3 .7 .9 .4 .3 ]

A matrix:

.3 .4 .8 .1 .2 .3

.5 .2 0 .1 .5 .2

.1 .1 .9 .2 .5 .3

.2 .4 .1 .7 .8 .5

.9 .9 .2 .5 .3 .5

.4 .1 .2 .7 .8 .2

.1 .2 .2 .5 .7 .2

[ 1.5 1.4 0.8 1.5 1.9 1.2 ]



A row vector:

[ .2 .6 .3 .7 .9 .4 .3 ]

A matrix:

.3 .4 .8 .1 .2 .3

.5 .2 0 .1 .5 .2

.1 .1 .9 .2 .5 .3

.2 .4 .1 .7 .8 .5

.9 .9 .2 .5 .3 .5

.4 .1 .2 .7 .8 .2

.1 .2 .2 .5 .7 .2

[ 1.5 1.4 0.8 1.5 1.9 1.2 ]



7 (rows) X 6 (columns) matrix

A row vector:

[ .2 .6 .3 .7 .9 .4 .3 ]

A matrix:

.3 .4 .8 .1 .2 .3

.5 .2 0 .1 .5 .2

.1 .1 .9 .2 .5 .3

.2 .4 .1 .7 .8 .5

.9 .9 .2 .5 .3 .5

.4 .1 .2 .7 .8 .2

.1 .2 .2 .5 .7 .2

[ 1.5 1.4 0.8 1.5 1.9 1.2 ]



7 (rows) X 6 (columns) matrix

NOT Commutative!

Multiplying a Matrix by a Matrix

• The Same Mantra: Rows by Columns

Rows by Columns

A 2 X 3 matrix A 3 X 2 matrix

1 2 3

1 2 3

1 2

1 2

1 2

Row 1 X Column 1


1 2 3

1 2 3

1 2

1 2

1 2

1

2

3

(proxy)

Row 1 Column 1

Row 1 X Column 1


1 2 3

1 2 3

1 2

1 2

1 2

1

2

3

Row 1 Column 1

Result = 6

Row 1 X Column 1


1 2 3

1 2 3

1 2

1 2

1 2

1

2

3

Row 1 Column 1

Result = 6

Place the result in row 1,

Row 1 X Column 1


1 2 3

1 2 3

1 2

1 2

1 2

1

2

3

Row 1 Column 1

Result = 6

Place the result in row 1, column 1

Row 1 X Column 1


1 2 3

1 2 3

1 2

1 2

1 2

1

2

3

Row 1 Column 1

Result = 6

Place the result in row 1, column 1 of a new matrix…

6

Row 1 X Column 2


1 2 3

1 2 3

1 2

1 2

1 2

1

2

3

Row 1 Column 2

Result = 12

6

Row 1 X Column 2


1 2 3

1 2 3

1 2

1 2

1 2

1

2

3

Row 1 Column 2

Result = 12

6 12Place the result in row 1, column 2 of the new matrix…

Row 2 X Column 1


1 2 3

1 2 3

1

2

3

Row 2

Column 1

Result = 6

6 12

1 2

1 2

1 2

Row 2 X Column 1


1 2 3

1 2 3

1

2

3

Row 2

Column 1

Result = 6

6 12

6

1 2

1 2

1 2

Place the result in row 2, column 1 of the new matrix…

Row 2 X Column 2


1 2 3

1 2 3

1

2

3

Row 2

Column 2

6 12

6

1 2

1 2

1 2

Result = 12

Row 2 X Column 2


1 2 3

1 2 3

1

2

3

Row 2

Column 2

6 12

6 12

1 2

1 2

1 2

Result = 12

Place the result in row 2, column 2 of the new matrix…

So…


1 2 3

1 2 3

6 12

6 12

1 2

1 2

1 2*

=

A 2 X 2 matrix

So…


1 2 3

1 2 3

6 12

6 12

1 2

1 2

1 2*

=

A 2 X 2 matrix

The result has the same number of rows as the first matrix…

So…


1 2 3

1 2 3

6 12

6 12

1 2

1 2

1 2*

=

A 2 X 2 matrix

The result has the same number of rows as the first matrix…

…and the same number of columns as the second.

…and…


1 2 3

1 2 3

6 12

6 12

1 2

1 2

1 2*

=

A 2 X 2 matrix

…and the number of columns in the first matrix…

…and…


1 2 3

1 2 3

6 12

6 12

1 2

1 2

1 2*

=

A 2 X 2 matrix

…and the number of columns in the first matrix…

…must be equal to the number of rows in the second.

This is basic (default) matrix

multiplication.

There’s other more complicated stuff, too.

You (probably) won’t need it for this class.

Documents

Prelude A pattern of activation in a NN is a vector A set of connection weights between units is a matrix Vectors and matrices have well-understood mathematical