Complexity Analysis

2

Complexity

The complexity of an algorithm quantifies the resources needed as a

function of the amount of input data size.

The resource measured is usually time (cpu cycles), or sometimes

space (memory).

Complexity is not an absolute measure, but a bounding function

characterizing the behavior of the algorithm as the size of the data

set increases.

3

Usefulness

It allows the comparison of algorithms for efficiency, and predicts their

behavior as data size increases.

A particular algorithm may take years to compute. If you know

beforehand, you won’t just wait for it to finish

4

Big-Oh notation indicates the family to which an algorithm belongs.

Formally, T(n) is O(f(n)) if there exist positive constants k and n0 such that

|T(n)| k |f(n)| for all n > n0.

Examples

n

T(n)

n

T(n)

linear quadratic

Complexity defined

5

Complexity Classes

There is a hierarchy of complexities.

Here’s some of the hierarchy:

O(1) < O(log n) < O(n) < O(n logn) < O(n2) < O(n3) < O(2n) < O(n!)

An algorithm with smaller complexity is normally preferable to one with larger

complexity.

Tuning an algorithm without affecting its complexity is usually not as good as

finding an algorithm with better complexity.

6

Importance of Complexity

consider each operation (step) takes 1s:

n log2n n log2n n2 2n

----------------------------------------------------------------------

2 1s 2s 4s 4s

16 4s 64s 256s 65ms

256 8s 2ms 65ms 4 x 1063 years

1024 10s 10ms 1sec 6 x 10294 years

7

But computers are getting faster, right ?

CPU speed 1.5 faster every 18 months

(True fact re: circuit density “Moore’s law” 1965)

8

Computers are getting faster, but …

consider each operation (step) takes 1s:

n log2n n log2n n2 2n

----------------------------------------------------------------------

2 1s 2s 4s 4s

16 4s 64s 256s 65ms

256 8s 2ms 65ms 4 x 1063 years

1024 10s 10ms 1sec 6 x 10294 years

NP-complete – class of problems best known alg. O(2n)

9

Importance of Complexity

consider each operation (step) takes 1s:

n = 1 million

Bubble sort will take O(n2) or ~1012 operations or 277 hours.

Quicksort, mergesort will take O(n logn) or ~107 operations or 10 sec.

Sequential search will take O(n) or 500000 comparisons (average).

Binary search will take O(log n) or 20 comparisons on average.

10

Importance of Complexity

Trivia: first binary search algo published in 1946

first correct binary search algo published in 1962

First tune the algorithm, then tune the code !

11

Deriving f(n) from T(n)

Reduce to the dominant term: As n becomes very large, what is the

approximation of T(n)?

E.g. O(n2 + 106n) = O(n2).

Eliminate multiplicative constants: It’s the characteristic “shape” of f(n) that

matters.

E.g. O(3n) = O(n) = O(n/3).

12

BubbleSort (non-recursive)

for (i = n; i >= 1; i--) { /* Line 1 */ for (j = 1; j < i; j++) { /* 2 */ if (a[j-1] > a[j]) { /* 3 */ temp = a[j-1]; /* 4 */ a[j-1] = a[j]; /* 5 */ a[j] = temp; /* 6 */ } }}

13

BubbleSort

for (i = n; i >= 1; i--) { /* Line 1 */ for (j = 1; j < i; j++) { /* 2 */ if (a[j-1] > a[j]) { /* 3 */ temp = a[j-1]; /* 4 */ a[j-1] = a[j]; /* 5 */ a[j] = temp; /* 6 */ } }}

Line 1: Executed n times.

Lines 2 & 3: Executed n + (n-1) + (n-2) + … + 2 + 1 = n(n + 1) / 2 times.

O((n2 + n)/2) = O(n2)

14

Factorial (recursive functions)

int fact (int n) { if (n == 0) return 1; return n * fact (n – 1);}

T(n) = T(n - 1) + 1

T(1) = 0

T(n) = 1+ T(n - 1) = 1 + 1 + T(n - 2) = 1 + 1 + 1 + … = n

O(n)

15

More Formally: Solutions for Recursion

Method 1: Induction

- guess solution (constant k and function f(n) such that

|T(n)| k |f(n)| for all n > n0

- prove by induction

Method 2: Iterate T(n) recurrence, then prove by induction

Method 3: Recursion trees

16

Recurrence Equations (Method 2)

MergeSort

T(n) - time for merge sort = ?

Calls itself recursively on the two halves (so 2T(n/2)

Merges the two halves in n steps

T(n) = n + 2T(n/2)

T(1) = 1

17

Recurrence Equations (Method 2)

T(n) – time for merge sort

T(n) = n + 2T(n/2) = n + 2 (n/2 + 2T(n/4)) =

= n + n + 4T(n/4) = n + n + 4(n/4 + 2T(n/8)) =

= n + n + n + 8(n/8 + 2T(n/16)) = … = n logn

18

Recurrence Trees: Method 3

int fibonacci (int n) { if (n <= 1) return 1; return fibonacci(n – 1) + fibonacci (n - 2);}

1 1 2 3 5 8 13 …