Introduction to Algorithms: Asymptotic Notationcs.boisestate.edu/.../NewSlides/AsymptoticNotation.pdfAsymptotic Notation that 0 f(n) cg(n) for all n n 0. O-notation (upper bounds):

Introduction to Algorithms: Asymptotic Notation

Why Should We Care? (Order of growth)

CS 421 - Analysis of Algorithms 2

Asymptotic Notation

0that 0 f(n) cg(n) for all n n .

O-notation (upper bounds):


Asymptotic Notation



CS 421 - Analysis of Algorithms 454 Fundamentals of the Analysis of Algorithm Efficiency

doesn'tmatter

nn0

cg(n)

t (n)

FIGURE 2.1 Big-oh notation: t (n) ∈ O(g(n)).

doesn'tmatter

nn0

cg(n)

t (n)

FIGURE 2.2 Big-omega notation: t (n) ∈ !(g(n)).

!-notationDEFINITION A function t (n) is said to be in !(g(n)), denoted t (n) ∈ !(g(n)), ift (n) is bounded below by some positive constant multiple of g(n) for all large n,

i.e., if there exist some positive constant c and some nonnegative integer n0 suchthat

t (n) ≥ cg(n) for all n ≥ n0.

The definition is illustrated in Figure 2.2.

Here is an example of the formal proof that n3 ∈ !(n2):

n3 ≥ n2 for all n ≥ 0,

i.e., we can select c = 1 and n0 = 0.

54 Fundamentals of the Analysis of Algorithm Efficiency

doesn'tmatter

nn0

cg(n)

t (n)

FIGURE 2.1 Big-oh notation: t (n) ∈ O(g(n)).

doesn'tmatter

nn0

cg(n)

t (n)

FIGURE 2.2 Big-omega notation: t (n) ∈ !(g(n)).

!-notationDEFINITION A function t (n) is said to be in !(g(n)), denoted t (n) ∈ !(g(n)), ift (n) is bounded below by some positive constant multiple of g(n) for all large n,

i.e., if there exist some positive constant c and some nonnegative integer n0 suchthat

t (n) ≥ cg(n) for all n ≥ n0.

The definition is illustrated in Figure 2.2.

Here is an example of the formal proof that n3 ∈ !(n2):

n3 ≥ n2 for all n ≥ 0,

i.e., we can select c = 1 and n0 = 0.

Asymptotic Notation




where c = 1, n0 = 2

Asymptotic Notation




where c = 1, n0 = 2functions, not values

funny, one-way equality

Asymptotic Notation



Asymptotic Notation



2.2 Asymptotic Notations and Basic Efficiency Classes 53

Indeed, the first two functions are linear and hence have a lower order of growththan g(n) = n2, while the last one is quadratic and hence has the same order ofgrowth as n2. On the other hand,

n3 ̸∈ O(n2), 0.00001n3 ̸∈ O(n2), n4 + n + 1 ̸∈ O(n2).

Indeed, the functions n3 and 0.00001n3 are both cubic and hence have a higherorder of growth than n2, and so has the fourth-degree polynomial n4 + n + 1.

The second notation, !(g(n)), stands for the set of all functions with a higheror same order of growth as g(n) (to within a constant multiple, as n goes to infinity).For example,

n3 ∈ !(n2),12n(n − 1) ∈ !(n2), but 100n + 5 ̸∈ !(n2).

Finally, "(g(n)) is the set of all functions that have the same order of growthas g(n) (to within a constant multiple, as n goes to infinity). Thus, every quadraticfunction an2 + bn + c with a > 0 is in "(n2), but so are, among infinitely manyothers, n2 + sin n and n2 + log n. (Can you explain why?)

Hopefully, this informal introduction has made you comfortable with the ideabehind the three asymptotic notations. So now come the formal definitions.

O-notationDEFINITION A function t (n) is said to be in O(g(n)), denoted t (n) ∈ O(g(n)),

if t (n) is bounded above by some constant multiple of g(n) for all large n, i.e., ifthere exist some positive constant c and some nonnegative integer n0 such that

t (n) ≤ cg(n) for all n ≥ n0.

The definition is illustrated in Figure 2.1 where, for the sake of visual clarity, n isextended to be a real number.

As an example, let us formally prove one of the assertions made in theintroduction: 100n + 5 ∈ O(n2). Indeed,

100n + 5 ≤ 100n + n (for all n ≥ 5) = 101n ≤ 101n2.

Thus, as values of the constants c and n0 required by the definition, we can take101 and 5, respectively.

Note that the definition gives us a lot of freedom in choosing specific valuesfor constants c and n0. For example, we could also reason that

100n + 5 ≤ 100n + 5n (for all n ≥ 1) = 105n

to complete the proof with c = 105 and n0 = 1.

Set Definition of Ο-notation


Ο(𝑔 𝑛 ) is the set of all functions with a SMALLER or the SAME order of growth as 𝑔 𝑛 .

Set Definition of Ο-notation


Ο(𝑔 𝑛 ) is the set of all functions with a SMALLER or the SAME order of growth as 𝑓(𝑛).

EXAMPLE: Ο(𝑛') = 5𝑛', 10𝑛- + 500, log𝑛, …

5𝑛' ∈ Ο(𝑛')Or put another way:

Ω-notation (lower bounds)

Asymptotic Notation


Ω-notation (lower bounds):


Asymptotic Notation




Asymptotic Notation




where c = 1, n0 = 16

Asymptotic Notation




Set Definition of Ω-notation


Ω(𝑔 𝑛 ) is the set of all functions with a LARGER or the SAME order of growth as 𝑔 𝑛 .

EXAMPLE: Ω(𝑛-) = 5𝑛-, 10𝑛' + 500, 𝑛- ∗ log 𝑛, …

5𝑛- ∈ Ω(𝑛-)Or put another way:


Θ-notation (tight bounds)Combine definitions of Ο and Ω:

𝑊𝑒𝑤𝑟𝑖𝑡𝑒:𝑓 𝑛 = Θ 𝑔 𝑛

𝑖𝑓𝑡ℎ𝑒𝑟𝑒𝑒𝑥𝑖𝑠𝑡𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡𝑠𝑐1 > 0, 𝑐2 > 0𝑎𝑛𝑑𝑛0 > 0

𝑠𝑢𝑐ℎ𝑡ℎ𝑎𝑡c1𝑔 𝑛 <= 𝑓 𝑛 <= 𝑐2𝑔(𝑛)


Θ-notation (tight bounds)Combine definitions of Ο and Ω:

𝑊𝑒𝑤𝑟𝑖𝑡𝑒:𝑓 𝑛 = Θ 𝑔 𝑛

𝑖𝑓𝑡ℎ𝑒𝑟𝑒𝑒𝑥𝑖𝑠𝑡𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡𝑠𝑐1 > 0, 𝑐2 > 0𝑎𝑛𝑑𝑛0 > 0

𝑠𝑢𝑐ℎ𝑡ℎ𝑎𝑡c1𝑔 𝑛 <= 𝑓 𝑛 <= 𝑐2𝑔(𝑛)

Θ-notation (tight bounds)






Upper bound



Upper bound

Lower bound



Upper bound

Lower bound


CS 421 - Analysis of Algorithms

EXAMPLE:

24


ο-notationO-notation like ≤.o-notation like <.𝑊𝑒𝑤𝑟𝑖𝑡𝑒: 𝑓 𝑛 = 𝜊 𝑔 𝑛

𝑓𝑜𝑟𝑎𝑛𝑦𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡𝑠𝑐 > 0,

𝑡ℎ𝑒𝑟𝑒𝑖𝑠𝑎𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡𝑛M > 0𝑠𝑢𝑐ℎ𝑡ℎ𝑎𝑡

0 < 𝑓(𝑛) < 𝑐𝑔(𝑛)


ο-notationO-notation like ≤.o-notation like <.𝑊𝑒𝑤𝑟𝑖𝑡𝑒: 𝑓 𝑛 = 𝜊 𝑔 𝑛



0 < 𝑓(𝑛) < 𝑐𝑔(𝑛)

EXAMPLE: n0 = 2/c2n2 = o(n3),

Set Definition of ο-notation


ο(𝑔 𝑛 ) is the set of all functions with a strictly SMALLER order of growth as 𝑔 𝑛 .

Set Definition of ο-notation


ο(𝑔 𝑛 ) is the set of all functions with a SMALLER order of growth as 𝑔 𝑛 .

EXAMPLE: ο(𝑛') = 10𝑛- + 500, log 𝑛, …

10𝑛- + 500 ∈ ο(𝑛')Or put another way:


ω-notationΩ-notation is like ≥.ω-notation is like >.𝑊𝑒𝑤𝑟𝑖𝑡𝑒: 𝑓 𝑛 = ω 𝑔 𝑛



0 < 𝑐𝑔(𝑛) < 𝑓(𝑛)


ω-notationΩ-notation is like ≥.ω-notation is like >.𝑊𝑒𝑤𝑟𝑖𝑡𝑒: 𝑓 𝑛 = ω 𝑔 𝑛



0 < 𝑐𝑔(𝑛) < 𝑓(𝑛)

EXAMPLE: n0 = 1+1/cn =ω(lg n),

Set Definition of ω-notation


ω(𝑓(𝑛)) is the set of all functions with a strictly LARGER order of growth as 𝑓(𝑛).

Set Definition of ω-notation


ω(𝑓(𝑛)) is the set of all functions with a strictly LARGER order of growth as 𝑓(𝑛).

EXAMPLE: ω(𝑛-) = 10𝑛' + 500, 𝑛- ∗ log𝑛, …

10𝑛' ∈ ω(𝑛-)Or put another way:

Properties: Asymptotic Notation

• Transitivity• Reflexivity• Symmetry• Transposition


TransitivityAssuming f(n) and g(n) are asymptotically positive:

𝑓(𝑛) = Θ(𝑔(𝑛))and

𝑔(𝑛) = Θ(ℎ(𝑛))implies

𝑓(𝑛) = Θ(ℎ(𝑛))Holds for Ο, Ω, ο, and ω relations as well.


ReflexivityAssuming f(n) is asymptotically positive:

𝑓(𝑛) = Ο(𝑓(𝑛))and

𝑓(𝑛) = Ω(𝑓(𝑛))and

𝑓(𝑛) = Θ(𝑓(𝑛))

DOES NOT hold for ο and ω relations.CS 421 - Analysis of Algorithms 35

SymmetryAssuming f(n) and g(n) are asymptotically positive:

𝑓(𝑛) = Θ(𝑔(𝑛))

iff (if, and only if,)

𝑔(𝑛) = Θ(𝑓(𝑛))


TransposeAssuming f(n) and g(n) are asymptotically positive:𝑓(𝑛) = Ο(𝑔(𝑛)) iff 𝑔(𝑛) = Ω(𝑓(𝑛))

and

𝑓(𝑛) = ο(𝑔(𝑛)) iff 𝑔(𝑛) = ω(𝑓(𝑛))


Analogy with numbers






Useful Property

𝑓1(𝑛) = Θ(𝑔1(𝑛))and

𝑓2 𝑛 = Θ(𝑔2(𝑛))implies

𝑓1 𝑛 + 𝑓2 𝑛 = Θ(max{𝑔1(𝑛), 𝑔2(𝑛)})

Holds for Ο, Ω, ο, and ω relations as well.


Using Limits to Compare Growth Rates


Though using Ο, Ω, ο, and ω indispensable for comparing growth rates of functions in the abstract, when comparing actual functions, convenient to

Using Limits to Compare Growth Rates


limV→X

𝑓(𝑛)𝑔(𝑛)

=0 - f(n) has smaller growth rate than g(n)

c - f(n) has same growth rate as g(n)

∞ - f(n) has larger growth rate than g(n)

• first two cases ⟹𝑓 𝑛 ∈ Ο 𝑔 𝑛• last two cases ⟹𝑓 𝑛 ∈ Ω(𝑔 𝑛 )• second case ⟹𝑓 𝑛 ∈ Θ(𝑔 𝑛 )

Growth Rates (Example 1)












L'Hôpital's rulehttps://en.wikipedia.org/wiki/L%27Hôpital%27s_rule



L'Hôpital's rulehttps://en.wikipedia.org/wiki/L%27Hôpital%27s_rule



Derivativeshttps://www.wyzant.com/resources/lessons/math/calculus/differentiation/list_of_derivatives

http://www.ripmat.it/mate/c/cf/cfdb.html







Stirling's approximation

https://en.wikipedia.org/wiki/Stirling%27s_approximation





Most Common Growth Rates


Class Name Examples1 Constant Only used in best-case efficiencies.log n logarithmic Result of cutting problem size by a

constant factor, like Binary Searchn linear Algorithms that scan a list of size n, like

Sequential, or Linear, Searchn log n n-log-n Divide-and-Conquer algorithms, like

Merge Sort and Quick Sortn2 quadratic Efficiencies with two embedded loops,

Bubble Sort and Insertion Sortn3 cubic Efficiencies with three embedded loops,

like many linear algebra algorithms2n exponential Algorithms that generate all sub-sets of an

n-element setn! factorial Algorithms that generate all permutations

of an n-element set

Macro SubstitutionConvention: A set in a formula represents an anonymous function in the set.

Macro substitution


Convention: A set in a formula represents an anonymous function in the set.

f(n) = n3 + O(n2) means

f(n) = n3 + h(n)for some

h(n) ∈ O(n2)

EXAMPLE:

59

Macro substitution


Convention: A set in a formula represents an anonymous function in the set.EXAMPLE:

60

Documents

Introduction to Algorithms: Asymptotic Notationcs.boisestate.edu/.../NewSlides/AsymptoticNotation.pdfAsymptotic Notation that 0 f(n) cg(n) for all n n 0. O-notation (upper bounds):