CS 2133: Data Structures Mathematics Review and Asymptotic Notation

CS 2133: Data Structures

Mathematics Review and

Asymptotic Notation

Arithmetic Series Review

1 + 2 + 3 + . . . + n = ?

Sn = a + a+d + a+2d + a+3d + . . . a+(n-1)dSn= 2a+(n-1)d + 2a+(n-2)d + 2a+(n-3)d + … + a

2Sn = 2a+(n-1)d + 2a+(n-1)d + 2a+(n-1)d . . . + 2a+(n-1)dSn = n/2[2a + (n-1)d]

consequently 1+2+3+…+n = n(n+1)/2

Problems

Find the sum of the following

1+3+5+ . . . + 121 = ?

The first 50 terms of -3 + 3 + 9 + 15 + …

1 + 3/2 + 2 + 5/2 + . . . 25=?

Geometric Series Review

1 + 2 + 4 + 8 + . . . + 2n

1 + 1/2 + 1/4 + . . . + 2-n

r

arara

nn

i

i

1

1

0

Theorem:

Sn= a + ar + ar2 + . . . + arn

rSn= ar + ar2 + . . . + arn + arn+1

Sn-rSn = a - arn+1 r

araS

n

n

1

1

What about the case where -1< r < 1 ?

Geometric Problems

?8

1

4

1

2

11

2

1

0

i

i

What is the sum of 3+9/4 + 27/16 + . . .

1/2 - 1/4 + 1/8 - 1/16 + . . .

Harmonic Series

577.0

ln

14

13

12

11

nH

nH

n

n

This is Eulers constants

Just an Interesting Question

What is the optimal base to use in the representation of numbers n?

Example: with base x we have _ _ _ _ _ _ _ _ _

1log nx

X values

slots

We minimize

xnc 1log2

Logarithm Review

Ln = loge is called the natural logarithmLg = log2 is called the binary logarithm

How many bits are required to represent the number n in binary

1log2 n

Logarithm Rules

The logarithm to the base b of x denoted logbx is defined to that number y such that

by = x

logbx > 0 if x > 1logbx = 0 if x = 1logbx < 0 if 0 < x < 1

logb(x1*x2) = logb x1 + logb x2

logb(x1/x2) = logb x1 - logb x2

logb xc = c logbx

Additional Rules

For all real a>0, b>0 , c>0 and n

logb a = logca / logc b logb (1/a) = - logb a

logb a = 1/ logab a logb n = n log

b a

Asymptotic Performance

In this course, we care most about the asymptotic performance of an algorithm. How does the algorithm behave as the problem

size gets very large? Running time Memory requirements

Coming up: Asymptotic performance of two search algorithms, A formal introduction to asymptotic notation

Input Size

Time and space complexity This is generally a function of the input size

E.g., sorting, multiplication How we characterize input size depends:

Sorting: number of input items Multiplication: total number of bits Graph algorithms: number of nodes & edges Etc

Running Time

Number of primitive steps that are executed Except for time of executing a function call most

statements roughly require the same amount of time y = m * x + b c = 5 / 9 * (t - 32 ) z = f(x) + g(y)

We can be more exact if need be

Analysis

Worst case Provides an upper bound on running time An absolute guarantee

Average case Provides the expected running time Very useful, but treat with care: what is “average”?

Random (equally likely) inputs Real-life inputs

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

What is the preconditionfor this loop?

Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}How many times will this loop execute?

Insertion Sort

Statement EffortInsertionSort(A, n) {

for i = 2 to n { c1n

key = A[i] c2(n-1)

j = i - 1; c3(n-1)

while (j > 0) and (A[j] > key) { c4T

A[j+1] = A[j] c5(T-(n-1))

j = j - 1 c6(T-(n-1))

} 0

A[j+1] = key c7(n-1)

} 0

}

T = t2 + t3 + … + tn where ti is number of while expression evaluations for the ith for loop iteration

Analyzing Insertion Sort

T(n) = c1n + c2(n-1) + c3(n-1) + c4T + c5(T - (n-1)) + c6(T - (n-1)) + c7(n-1)

= c8T + c9n + c10

What can T be? Best case -- inner loop body never executed

ti = 1 T(n) is a linear function Worst case -- inner loop body executed for all

previous elements ti = i T(n) is a quadratic function T=1+2+3+4+ . . . n-1 + n = n(n+1)/2

Analysis

Simplifications Ignore actual and abstract statement costs Order of growth is the interesting measure:

Highest-order term is what counts Remember, we are doing asymptotic analysis As the input size grows larger it is the high order term that

dominates

Upper Bound Notation

We say InsertionSort’s run time is O(n2) Properly we should say run time is in O(n2) Read O as “Big-O” (you’ll also hear it as “order”)

In general a function f(n) is O(g(n)) if there exist positive constants c and n0

such that f(n) c g(n) for all n n0

Formally O(g(n)) = { f(n): positive constants c and n0 such that

f(n) c g(n) n n0

Big O example

Show using the definition that 5n+4 O(n)Where g(n)=nFirst we must find a c and an n0

We now need to show that f(n) c g(n) for every n n0

clearly 5n + 5 6n whenever n 6

Hence c=6 and n0=6 satisfy the requirements.

Insertion Sort Is O(n2)

Proof Suppose runtime is an2 + bn + c

If any of a, b, and c are less than 0 replace the constant with its absolute value

an2 + bn + c (a + b + c)n2 + (a + b + c)n + (a + b + c) 3(a + b + c)n2 for n 1 Let c’ = 3(a + b + c) and let n0 = 1

Question Is InsertionSort O(n3)? Is InsertionSort O(n)?

Big O Fact

A polynomial of degree k is O(nk) Proof:

Suppose f(n) = bknk + bk-1nk-1 + … + b1n + b0

Let ai = | bi |

f(n) aknk + ak-1nk-1 + … + a1n + a0

ki

kk

i

ik cnan

n

nan

Lower Bound Notation

We say InsertionSort’s run time is (n) In general a function

f(n) is (g(n)) if positive constants c and n0 such that 0 cg(n) f(n) n n0

Proof: Suppose run time is an + b

Assume a and b are positive (what if b is negative?) an an + b

Asymptotic Tight Bound

A function f(n) is (g(n)) if positive constants c1, c2, and n0 such that

c1 g(n) f(n) c2 g(n) n n0

Theorem f(n) is (g(n)) iff f(n) is both O(g(n)) and (g(n)) Proof: someday

Notation

(g) is the set of all functions f such that there exist positive constants c1, c2, and n0 such that

0 c1g(n) f(n) c2 g(n) for every n > nc

c1g(n)

f(n)

c2 g(n)

Growth Rate Theorems1. The power n is in O(n) iff (with ,>0) and n is in o(n) iff

2. logbn o(n ) for any b and

3. n o(cn) for any >0 and c>1

4. logbn O(logbn) for any a and b

5. cn O(dn) iff cd and cn o(dn) iff c<d

6. Any constant function f(n) =c is in O(1)

Big O Relationships

1. o(f) O(f)

2. If fo(g) then O(f) o(g)

3. If f O(g) then o(f) o(g)

4. If f O(g) then f(n) + g(n) O(g)

5. If f O(f `) and g O(g`) then

f(n)* g(n) O(f `(n) * g`(n))

Theorem: log(n!)(nlogn)Case 1 nlogn O(log(n!))

log(n!) = log(n*(n-1)*(n-2) * * * 3*2*1)

= log(n*(n-1)*(n-2)**n/2*(n/2-1)* * 2*1

=> log(n/2*n/2* * * n/2*1 *1*1* * * 1)

= log(n/2)n/2 = n/2 log n/2 O(nlogn)

Case 2 log(n!) O(nlogn)

log(n!) = logn + log(n-1) + log(n-2) + . . . Log(2) + log(1)

< log n + log n + log n . . . + log n

= nlogn

The Little o Theorem: If log(f)o(log(g)) andlim g(n) =inf as n goes to inf then f o(g)

Note the above theorem does not apply to big O for log(n2) O(log n) but n2 O(n)

Application: Show that 2n o(nn)Taking the log of functions we have log(2n)=nlog22and log( nn) = nlog2n.

Hence

2log

log

2log

loglimlim

n

n

nn

nn

Implies that 2n o(nn)

Theorem: )(lg non

n

n

n

nnn 2ln

lnlim

lglim

n

nn

lnlim

2ln

1

02

lim2ln

1

nn

)2(1

/1lim

2ln

1

n

nn

Practical Complexity

0

250

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

f(n) = n

f(n) = log(n)

f(n) = n log(n)

f(n) = n 2̂

f(n) = n 3̂

f(n) = 2 n̂

Practical Complexity

0

500

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

f(n) = n

f(n) = log(n)

f(n) = n log(n)

f(n) = n 2̂

f(n) = n 3̂

f(n) = 2 n̂

Practical Complexity

0

1000

1 3 5 7 9 11 13 15 17 19

f(n) = n

f(n) = log(n)

f(n) = n log(n)

f(n) = n 2̂

f(n) = n 3̂

f(n) = 2 n̂

Practical Complexity

0

1000

2000

3000

4000

5000

1 3 5 7 9 11 13 15 17 19

f(n) = n

f(n) = log(n)

f(n) = n log(n)

f(n) = n 2̂

f(n) = n 3̂

f(n) = 2 n̂

Other Asymptotic Notations

A function f(n) is o(g(n)) if positive constants c and n0 such that

f(n) < c g(n) n n0

A function f(n) is (g(n)) if positive constants c and n0 such that

c g(n) < f(n) n n0

Intuitively, o() is like < O() is like

() is like > () is like

() is like =

Comparing functions

Definition: The function f is said to dominate g if f(n)/g(n)increases without bound as n increases without bound.

i.e. for any c>0 there exist n0>0 such that f(n)> c g(n)for every n>n0

)(

)(lim

ng

nfn

1log2!2 222

nnn nn

Little o Complexity

o(g) is the set of all functions that are dominated by g, i.e. The set of all f such that for every c>0 there exist nc>0such that f(n)c g(n) for every n > nc

Up Next

Solving recurrences Substitution method Master theorem