CS 401

Multiplication / FFTXiaorui Sun

1

StuffSolution of homework 2 will be posted on course website by the end of today

Additional office hour: Monday 2pm–4pm at SEO 1241

Midterm review: Oct 16

Midterm exam: October 18 in class1pm-1:50pm, SES 138You may use a sheet with notes on both sides, but not textbook and any other paper materialsYou may use a calculator, but not any device with transmitting functions, especially ones that can access the wireless or the Internet.Be concise with your solution, always give your high level idea.

Fast Fourier Transform

Polynomials: Coefficient RepresentationPolynomial. [coefficient representation]

Add: O(n) arithmetic operations.

Evaluate: O(n) using Horner's method.

Multiply (convolve): O(n2) using brute force.

!!

A(x) = a0 + a1x + a2x2 +"+ an-1x

n-1

!!

B(x) = b0 +b1x+b2x2 +"+ bn-1x

n-1

!!

A(x)+ B(x) = (a0 +b0 )+ (a1+b1)x +"+ (an-1 +bn-1)xn-1

!!

A(x)= a0 + (x (a1+ x (a2 +"+ x (an-2 + x (an-1))"))

A(x)´ B(x) = ci xi

i =0

2n-2å , where ci = a j bi- j

j =0

i

å4

Polynomials: Point-Value RepresentationPolynomial. [point-value representation]

Add: O(n) arithmetic operations.

Multiply: O(n), but need 2n-1 points.

Evaluate: O(n2) using Lagrange's formula.

!!

A(x) : (x0, y0 ),", (xn-1, yn-1) B(x) : (x0, z0 ),", (xn-1, zn-1)

!!

A(x)+ B(x) : (x0, y0 + z0 ),", (xn-1, yn-1 + zn-1)

A(x) = yk

(x - x j )j¹kÕ

(xk - x j )j¹kÕk=0

n-1å

0 0 0 2n-2 2 2 2 2( ) ( ) : (x , ) , , (x , )n nA x B x y z y z- -´ ´ ´!

5

Converting Between Two

RepresentationsTradeoff. Fast evaluation or fast multiplication. We want

both!

Goal. Make all ops fast by efficiently converting between

two representations.

Coefficient

Representation

O(n2)

Multiply

O(n)

Evaluate

Point-value O(n) O(n2)

!!

a0, a1,", an-1 !!

(x0, y0 ),", (xn-1, yn-1)coefficient

representation

point-valuerepresentation

FFT: O(n log n)

Inverse FFT: O(n log n)

6

Coefficient to Point-Value Representation: Intuition

Coefficient to point-value. Given a polynomial a0 + a1 x + ... + an-1 xn-1, evaluate it at n distinct points x0, ... , xn-1.Divide. Break polynomial up into even and odd powers.

A(x) = a0 + a1x + a2x2 + a3x3 + a4x4 + a5x5 + a6x6 + a7x7.Aeven(x) = a0 + a2x + a4x2 + a6x3.Aodd (x) = a1 + a3x + a5x2 + a7x3.A(-x) = Aeven(x2) + x Aodd(x2).A(-x) = Aeven(x2) - x Aodd(x2).

Intuition. Choose two points to be ±1.A(-1) = Aeven(1) + 1 Aodd(1). A(-1) = Aeven(1) - 1 Aodd(1).

Can evaluate polynomial of degree £ nat 2 points by evaluating two polynomials of

degree £ ½n at 1 point.

7

Coefficient to Point-Value Representation: Intuition

Coefficient to point-value. Given a polynomial a0 + a1 x + ... + an-1 xn-1, evaluate it at n distinct points x0, ... , xn-1.Divide. Break polynomial up into even and odd powers.

A(x) = a0 + a1x + a2x2 + a3x3 + a4x4 + a5x5 + a6x6 + a7x7.Aeven(x) = a0 + a2x + a4x2 + a6x3.Aodd (x) = a1 + a3x + a5x2 + a7x3.A(-x) = Aeven(x2) + x Aodd(x2).A(-x) = Aeven(x2) - x Aodd(x2).

Intuition. Choose four points to be ±1, ±i.A(-1) = Aeven(-1) + 1 Aodd( 1). A(-1) = Aeven(-1) - 1 Aodd(-1).A(-i) = Aeven(-1) + i Aodd(-1). A(-i) = Aeven(-1) - i Aodd(-1).

Can evaluate polynomial of degree £ nat 4 points by evaluating two polynomials of

degree £ ½n at 2 points.8

Fast Fourier TransformCoefficient to point-value. Given a polynomial a0 + a1 x + ... + an-1 xn-1, evaluate it at n distinct points x0, ... , xn-1.

Key idea: choose xk = wk where w is principal nth root of unity.

!!

y0

y1y2

y3

"yn-1

é

ë

ê ê ê ê ê ê ê

ù

û

ú ú ú ú ú ú ú

=

1 1 1 1 # 11 w1 w2 w3 # wn-1

1 w2 w4 w6 # w2(n-1)

1 w3 w6 w9 # w3(n-1)

" " " " $ "1 wn-1 w2(n-1) w3(n-1) # w(n-1)(n-1)

é

ë

ê ê ê ê ê ê ê

ù

û

ú ú ú ú ú ú ú

a0

a1

a2

a3

"an-1

é

ë

ê ê ê ê ê ê ê

ù

û

ú ú ú ú ú ú ú

Fourier matrix Fn 9

Roots of UnityDef. An nth root of unity is a complex number x such that xn

= 1.Fact. The nth roots of unity are: w0, w1, …, wn-1 where w = e 2p i / n.Pf. (wk)n = (e 2p i k / n) n = (e p i ) 2k = (-1) 2k = 1.Fact. The ½nth roots of unity are: v0, v1, …, vn/2-1 where v = e 4p i / n.Fact. w2 = v and (w2)k = vk.

w0 = v0 = 1

w1

w2 = v1 = i

w3

w4 = v2 = -1

w5

w6 = v3 = -i

w7

n = 8

10

Fast Fourier TransformGoal. Evaluate a degree n-1 polynomial A(x) = a0 + .... + an-1 xn-1 at its nth roots of unity: w0, w1, …, wn-1.

Divide. Break polynomial up into even and odd powers.Aeven(x) = a0 + a2x + a4x2 + … + an/2-2 x(n-1)/2.

Aodd (x) = a1 + a3x + a5x2 + … + an/2-1 x(n-1)/2.

A(x) = Aeven(x2) + x Aodd(x2).

Conquer. Evaluate degree Aeven(x) and Aodd(x) at the ½nth

roots of unity: v0, v1, …, vn/2-1.

Combine. A(wk+n) = Aeven(vk) + wk Aodd(vk), 0 £ k < n/2

A(wk+n/2) = Aeven(vk) - wk Aodd(vk), 0 £ k < n/2

wk+n/2 = -wk

vk = (wk)2 = (wk+n/2)2

11

fft(n, a0,a1,…,an-1) {if (n == 1) return a0

(e0,e1,…,en/2-1) ¬ FFT(n/2, a0,a2,a4,…,an-2)(d0,d1,…,dn/2-1) ¬ FFT(n/2, a1,a3,a5,…,an-1)

for k = 0 to n/2 - 1 {wk ¬ e2pik/n

yk+n/2 ¬ ek + wk dkyk+n/2 ¬ ek - wk dk

}

return (y0,y1,…,yn-1)}

FFT Algorithm

12

FFT SummaryTheorem. FFT algorithm evaluates a degree n-1 polynomial at each of the nth roots of unity in O(n log n) steps.

Running time. T(2n) = 2T(n) + O(n) Þ T(n) = O(n log n).

assumes n is a power of 2

!!

a0, a1,", an-1 !!

(w0, y0 ),", (wn-1, yn-1)

O(n log n)

coefficientrepresentation

point-valuerepresentation

13

Recursion Treea0, a1, a2, a3, a4, a5, a6, a7

a1, a3, a5, a7a0, a2, a4, a6

a3, a7a1, a5a0, a4 a2, a6

a0 a4 a2 a6 a1 a5 a3 a7

"bit-reversed" order

000 100 010 110 001 101 011 111

perfect shuffle

14

Point-Value to Coefficient Representation: Inverse DFT

Goal. Given the values y0, ... , yn-1 of a degree n-1 polynomial at the n points w0, w1, …, wn-1, find unique polynomial a0 + a1 x + ... + an-1 xn-1 that has given values at given points.

Inverse DFT

!!

a0

a1

a2

a3

"an-1

é

ë

ê ê ê ê ê ê ê

ù

û

ú ú ú ú ú ú ú

=

1 1 1 1 # 11 w1 w2 w3 # wn-1

1 w2 w4 w6 # w2(n-1)

1 w3 w6 w9 # w3(n-1)

" " " " $ "1 wn-1 w2(n-1) w3(n-1) # w(n-1)(n-1)

é

ë

ê ê ê ê ê ê ê

ù

û

ú ú ú ú ú ú ú

-1

y0

y1y2

y3

"yn-1

é

ë

ê ê ê ê ê ê ê

ù

û

ú ú ú ú ú ú ú

Fourier matrix inverse (Fn)-1 15

Claim. Inverse of Fourier matrix is given by following formula.

Consequence. To compute inverse FFT, apply same algorithm but use w-1 = e -2p i / n as principal nth root of unity (and divide by n).

!!

Gn = 1n

1 1 1 1 " 11 w-1 w-2 w-3 " w-(n-1)

1 w-2 w-4 w-6 " w-2(n-1)

1 w-3 w-6 w-9 " w-3(n-1)

# # # # $ #1 w-(n-1) w-2(n-1) w-3(n-1) " w-(n-1)(n-1)

é

ë

ê ê ê ê ê ê ê

ù

û

ú ú ú ú ú ú ú

Inverse FFT

16

Inverse FFT: Algorithm

ifft(n, a0,a1,…,an-1) {if (n == 1) return a0

(e0,e1,…,en/2-1) ¬ FFT(n/2, a0,a2,a4,…,an-2)(d0,d1,…,dn/2-1) ¬ FFT(n/2, a1,a3,a5,…,an-1)

for k = 0 to n/2 - 1 {wk ¬ e-2pik/n

yk+n/2 ¬ (ek + wk dk)yk+n/2 ¬ (ek - wk dk)

}

return (y0,y1,…,yn-1)}

17Note. Need to divide the final result by n

Inverse FFT SummaryTheorem. Inverse FFT algorithm interpolates a degree n-1 polynomial given values at each of the nth roots of unity in O(n log n) steps.

assumes n is a power of 2

!!

a0, a1,", an-1 !!

(w0, y0 ),", (wn-1, yn-1)

O(n log n)

coefficientrepresentation

O(n log n) point-valuerepresentation

18

Polynomial MultiplicationTheorem. Can multiply two degree n-1 polynomials in O(n log n) steps.

!!

a0, a1,", an-1b0, b1,", bn-1 !!

c0, c1,", c2n-2

!!

A(x0),", A(x2n-1)B(x0 ),", B(x2n-1) !!

C(x0),C(x1),",C(x2n-1)O(n)

point-value multiplication

O(n log n)FFT inverse FFT O(n log n)

coefficientrepresentation coefficient

representation

Problem 5Given two sorted arrays of n elements, find the n-thsmallest elementBy query, e.g. what is the 3-rd smallest element of first

array?

1 3 4 7 8 10 11 405 9 13 17 18 20 31 33

Problem 5 hintBinary search: find an element in a sorted array in O(log n) time

SolutionSearch(", $%&, ℎ"(ℎ): find n-th largest smallest element (in the merged array) if it is in array " with value ≥ +[", $%&]and ≤ +[/, ℎ"(ℎ]

Search(", $%&, ℎ"(ℎ)If ℎ"(ℎ < $%& return nullset 1"2 = ⌊($%& + ℎ"(ℎ) / 2⌋if +[",1"2] is n-th smallest element, return +[",1"2]If +[",1"2] is smaller than n-th smallest element

return Search(", $%&,1"2 − 1)else

return Search(",1"2 + 1, ℎ"(ℎ)

SolutionHow to determine if ![#,%#&] is n-th smallest element?

Query:• %#&-th element of ![#]• () − %#&)-th and () − %#& + 1)-th element of ![3 − #]

1. If ! #,%#& ≥ ![3 − #, ) − %#&] and ![#,%#&] <= ![3 −# ) −%#& + 1] then ![#,%#&] is the solution

2. If ![#,%#&] < ![3 − #, ) − %#&], then ![#,%#&] is greater than solution

3. If ![2,%#&] > ![3 − #, ) − %#& + 1], then ![#,%#&] is smaller than solution

Summary of Divide and Conquer Divide: We reduce a problem to several subproblems.

each sub-problem is at most a constant fraction of the size of the original problem

Conquer: Recursively solve each subproblem

Combine: Merge the solutions

Examples:• Mergesort, Binary Search, Closest Point Pair, Integer

Multiplication, Matrix Multiplication, FFTLo

g n

leve

ls

n

n/2n/2

n/4

How to solve problems by D&CDefine subproblems• Sometimes, subproblems are same to the original

problem (just input size is smaller), e.g. Mergesort• Sometimes, need to elaborate the subproblems to allow

solving subproblem recursively, e.g. Counting Inversions

Algorithm design: how to obtain solution of a subproblem from solutions of smaller subproblems?

Correctness proof: show the combine algorithm is correct• Given solutions of smaller subproblems, combine

algorithm give the solution of the right answer of original subproblem

25

Running TimeWrite down recurrence relation

! " = $ ! "% + Θ("))

Master theorem: Suppose ! " = $ ! +, + Θ(")) for all

" > %. Then,

• If $ < %) then ! " = Θ ")

• If $ = %) then ! " = Θ ")log "

• If $ > %) then ! " = Θ "34567