Algorithms and Theory of Computing Lecture 2: Big-O

Algorithms and Theory of Computing

Lecture 2: Big-O NotationGraph Algorithms

Xiaohui Bei

MAS 714

August 13, 2021

Nanyang Technological University MAS 714 August 13, 2021 1 / 22

Basics of Algorithm Analysis


What is a good algorithm?

All Rights Reserved http://www.cartoonbank.com


Algorithm Performance

EfficiencyWhat qualifies as an efficient algorithm?

Most important factor: running time.Many possibilities: n, n logn, n2, n3, n100, 2n, n!, etc...

Brute force: enumerate every possible solutions that check their validity.Usually takes at least 2n time, is not considered as efficient.


Running Time Examples


Polynomial Running Time

Efficient AlgorithmsAn algorithm called efficient if it has a polynomial running time.

What is so good about polynomial running time?Robust, mathematically sound.When input size doubles, the running time only increases by some constant factor C.Works well in practice.

Drawbacks:Polynomials with large constants/exponents are not that practical.20n100 or n1+0.02 lnn?


Type of Analyses

Running TimeHow to define the running time of an algorithm?

Worst-case. Running time guarantee for any input of size n.

Average-case. Expected running time for a random input of size n.

Probabilistic. Expected running time of a randomized algorithm.


O,Ω, and Θ

Let T, f are two monotone increasing functions that maps from N to R+.

Asymptotic Upper BoundsWe say T(n) = O(f(n)) if there exists constants c > 0 and n0 ≥ 0, such that for all n ≥ n0,we have T(n) ≤ c · f(n).

ExamplesT(n) = 1000n+ 100 =⇒ T(n) = O(n)

I set c = 1001 and n0 = 100

T(n) = pn2 + qn+ r for constants p, q, r > 0 =⇒ T(n) = O(n2)I set c = p+ q+ r and n0 = 1I also correct to say T(n) = O(n3)


Some Remarks

Equals sign. O(f(n)) is a set of functions, but computer scientists often writeT(n) = O(f(n)) instead of T(n) ∈ O(f(n)).

I Consider f(n) = 5n3 and g(n) = 3n2, we write f(n) = O(n3) = g(n), but it does not meanf(n) = g(n).

Domain. The domain of f(n) is typically the natural numbers 0, 1, 2, . . ..I Sometimes we restrict to a subset of the natural numbers.

Other times we extend to the reals.

Nonnegative functions. When using big-O notation, we assume that the functionsinvolved are (asymptotically) nonnegative.


O,Ω, and Θ


Asymptotic Lower BoundsWe say T(n) = Ω(f(n)) if there exists constants ε > 0 and n0 ≥ 0, such that for all n ≥ n0,we have T(n) ≥ ε · f(n).

ExamplesT(n) = pn2 + qn+ r for constants p, q, r > 0 =⇒ T(n) = Ω(n2)

I set ε = p and n0 = 1I also correct to say T(n) = Ω(n)

Meaningful Statement. Any compare-based sorting algorithm requires Ω(n logn) compares inthe worst case.Meaningless Statement. Any compare-based sorting algorithm requires O(n logn) compares inthe worst case.


O,Ω, and Θ


Asymptotic Tight BoundsWe say T(n) = Θ(f(n)) if there exists constants c1, c2 > 0 and n0 ≥ 0, such that for alln ≥ n0, we have c1 · f(n) ≤ T(n) ≤ c2 · f(n).

Alternative definitionT(n) = Θ(f(n)) if T(n) is both O(f(n)) and also Ω(f(n)).

ExamplesT(n) = pn2 + qn+ r for constants p, q, r > 0 =⇒ T(n) = Θ(n2)

I set c1 = p, c2 = p+ q+ r and n0 = 1I T(n) is neither Θ(n) nor Θ(n3)


Properties of Asymptotic Growth Rates

1 If f = O(g) and g = O(h), then f = O(h).

2 If f = Ω(g) and g = Ω(h), then f = Ω(h).

3 If f = Θ(g) and g = Θ(h), then f = Θ(h).

4 If f = O(h) and g = O(h), then f+ g = O(h).

5 If g = O(f), then f+ g = Θ(f).

6 If limn→∞ f(n)

g(n)= c > 0, then f = Θ(g).


Some Common Functions

1 Polynomials. Let f be a polynomial of degree d, in which the coefficient ad is positive.Then f = Θ(nd).

I Asymptotic rate of growth is determined by their “high-order term”.

Polynomial-Time AlgorithmA polynomial-time algorithm is one with running time O(nd) for some constant d.

2 Logarithms. For every a, b > 1 and every x > 0, we have loga n = Θ(logb n) = O(nx).I No need to specify base (assuming it’s a constant).I Logarithms are always better than polynomials.

3 Exponentials. For every r > 1 and every d > 0, we have nd = O(rn).I Polynomials are always better than exponentials.


More Notations

Asymptotic Smaller

We say T(n) = o(f(n)) if limn→∞ T(n)

f(n)= 0.

Asymptotic Larger

We say T(n) = ω(f(n)) if limn→∞ f(n)

T(n)= 0.


O,Ω, and Θ with multiple variables

Asymptotic Upper BoundsWe say T(m,n) = O(f(m,n)) if there exists constants c > 0,m0 ≥ 0 and n0 ≥ 0, such thatfor all m ≥ m0 and n ≥ n0, we have T(m,n) ≤ c · f(m,n).

Similar definitions for Ω and Θ.

ExamplesT(m,n) = 32mn2 + 17mn+ 32n3

T(m,n) is both O(mn2 + n3) and O(mn3).T(m,n) is neither O(n3) nor O(mn2).


Data Structures

We discussed algorithms in peusdocode, but how the data will be represented in an actualimplementation of the algorithm?

Data Structures: a particular way of organizing data such that it can be used efficiently inan algorithm.Appropriately designed data structures can help to give more efficient algorithms.Data structure in “Number Addition”: arrays.

Program = Algorithm + Data Structure


Common Data Structures

Elementary Data StructuresI ArraysI Linked ListsI Stacks and Queues

Priority QueuesHash TablesBinary Search Treesmany others


Graph Algorithms


Graphs

One of the most important combinatoric object in Computer Science, Optimization,Combinatorics.

Figure: Seven Bridges of Königsberg


Graphs

GraphA graph G consists of a set V of vertices and a set E ofedges

Directed Graph: Each edge e ∈ E is an ordered pair(u, v) for some u, v ∈ V.Undirected Graph: Each edge e ∈ E is an unorderedpair u, v for some u, v ∈ V.

1

23

4

5

6

ExampleIn the above undirected graph G = (V, E):

V = 1, 2, 3, 4, 5, 6

E = 1, 2, 1, 3, 2, 3, 2, 4, 3, 4, 3, 5, 4, 5, 4, 6


Applications

Transportation NetworksThe map of routes served by an airline carrier/railwaycompany.

vertices are airports/train stations;an edges from u to v if there is a flight/railwaytrack between them.

Information NetworksThe World Wide Web can be viewed as a directedgraph.

vertices are Web pages;an edge from u to v if u has a hyperlink to v.

Social NetworksA collection of people who interact with each otherforms an undirected graph.

vertices are peoplean edge joining u and v if they are friends.

Dependency NetworksA directed graph that captures the interdependenciesamong a collection of objects, e.g., in an university:

vertices are courses offeredan edge from u to v if u is a prerequisite for v.


Notation and Convention

usually use n = |V | and m = |E|

u and v are the end points of an edge e = u, v

multi-graphs may haveI loops which are edges (u, u)I multi-edges which are different edges between same pair of vertices

in most of this class we only consider simple graphs


Documents

Algorithms and Theory of Computing Lecture 2: Big-O