1

EE 559 MIDTERM EXAM 3/9/2011 Jenkins 3:30 - 4:45 PM PST This is last year’s midterm exam. The ground rules are similar to this year’s, except last year had no calculators. Instructions 1 hour 15 minutes. Work all four problems. Show your work and reasoning for all problems to maximize your partial credit, and indicate your answers clearly. 100 points possible. This is exam is closed book. You may use two formula sheets (each up to standard size 8.5”x11”, two-sided). No other materials are permitted. No computers and no calculators. Please do your work on the pages in this exam; attach additional sheets of your work if needed. You may do scratch work on this exam or separately (your choice); but of course be sure to turn in (and clearly label) all work you want graded. Please note that where a problem asks you to “find…”, “state…”, or “give…” something, you may use any valid method (from any starting point) to obtain the result. On the other hand, where a problem asks you to “prove…”, “derive…”, or “show…” something, you are to develop the result from a more fundamental starting point, showing the steps needed to get to the result. Good luck. This exam package includes 12 pages, some of which are blank.

2

Please do not write on this page. Problem 1 begins on the next page.

Please do not write in this box

1.

2.

3.

4.

Total

3

Problem 1 (30 points) In a 3-class problem you are given 4 prototypes as follows:

(1,1) !S1

("1,1), (1,"1) !S2

("1,"1) !S3

Plot the points on a graph in 2-D feature space. In the following parts you will consider different methods for multiclass decisions.

(a) For the pairwise class comparison method, discriminant function gkj x( ) is used to

compare Sk with S j , and the decision rule is:

x !Sk iff gkj x( ) > 0 "j # k

with gkj x( ) = !g jk x( ) . Are the prototypes separable using this method, with linear

functions gkj x( )? If so, prove it by separating them: draw the decision boundaries,

clearly show and label the decision regions, and give expressions for the 3 discriminant functions g12, g23, g13 . If not, justify why not.

(b) For the “one vs. rest” method, discriminant function gk x( ) is used to compare Sk

!

with Sk! , and the decision rule is:

x !Sk iff gk x( ) > 0 and g j x( ) < 0 "j # k .

Are the prototypes separable using this method, with linear functions gk x( )? If so,

prove it by separating them: draw the decision boundaries, clearly show and label the decision regions, and give expressions for the 3 discriminant functions. If not, justify why not.

(c) For the maximal value method, using discriminant functions gk x( ) , the decision

rule is:

x !Sk iff gk x( ) > g j x( ) "j # k .

Are the prototypes separable using this method, with linear functions gk x( )? If so,

prove it by separating them: draw the decision boundaries, clearly show and label the decision regions, and give expressions for the 3 discriminant functions. If not, justify why not.

4

5

6

Problem 2 (30 points) All parts of this problem work in augmented space and use reflected prototypes. Simplify your answers as much as possible.

(a) (10 points) State the MSE criterion function, J MSE w( ) . Be sure to clearly define

your quantities (matrices, etc.). Starting from your MSE criterion function, derive a one-pass training algorithm based on minimization by setting the gradient equal to 0.

Express your result in matrix (or matrix-vector) form. Assume that YT Y( ) is

nonsingular.

Hint: !x M x + a

2= 2M T M x + a( )

(b) (12 points) Consider instead the criterion function:

J w( ) = J MSE w( ) + 1

2! w

2

in which ! is a constant parameter. Describe in words the effect of the new term of J. Starting from

J w( ) , derive a one-pass training algorithm based on minimization by setting the gradient equal to 0. Express your result in matrix (or matrix-vector) form.

(c) (8 points) [Can be answered even if you didn’t finish part (b).] For the criterion function of (b), derive a batch-update gradient descent algorithm. Describe in words

the effect of the criterion-function term 12! w

2 on the iterative algorithm.

7

8

Problem 3 (28 points) Consider a support vector machine classifier with 2 classes and a linear, identity mapping from the original feature space to the new feature space. (a) (8 points) Given the following 4 prototypes in 1-D. nonaugmented feature space:

S1 : y1(1) = !2, y2

(1) = 2

S2 : y1(2) = 3, y1

(2) = 5

Find the SVM decision boundary: state it (give an equation) and plot it. No need to derive the result if you can deduce it from known characteristics of SVM classifiers.

Is the decision boundary completely determined (i.e., is there only one possible SVM boundary given the prototypes)? Briefly justify.

(b) (10 points) Given the following 4 prototypes in 2-D. nonaugmented feature space:

S1 : y1(1) = !2,0( )T , y2

(1) = 2,0( )T

S2 : y1(2) = 3,0( )T , y1

(2) = 5,0( )T

Find the SVM decision boundary: state it (give an equation) and plot it. No need to derive the result if you can deduce it from known characteristics of SVM classifiers.

Does the second feature ( x2 ) add any useful information?

Is the decision boundary completely determined (i.e., is there only one possible SVM boundary given the prototypes)? Briefly justify.

(c) (10 points) Given the following 4 prototypes in N-D. nonaugmented feature space:

S1 : y1(1) = !2,0,0,!,0( )T , y2

(1) = 2,0,0,!,0( )T

S2 : y1(2) = 3,0,0,!,0( )T , y1

(2) = 5,0,0,!,0( )T

Find the SVM decision boundary: state it (give an equation). No need to derive the result if you can deduce it from known characteristics of SVM classifiers.

Do the additional features ( x3 to xN ) add any useful information?

9

Is the decision boundary completely determined (i.e., is there only one possible SVM boundary given the prototypes)? Briefly justify.

Do the additional features seem likely to lead to overfitting in this case (i.e., are there too many degrees of freedom available in determining the decision boundary)? Briefly justify.

10

11

Problem 4 (12 points) You are given the following criterion function in a 2-class training problem:

J w( ) =

wT !ywi=1

P

!

in which the space is augmented, the prototypes are reflected, and the sum extends over all prototypes. Interpret J in words and geometrically. For the training problem, should J be minimized or maximized? Would the extremal point likely give a good decision boundary? Briefly justify. (No need to derive a learning algorithm or to carry out the optimization.)

12