19 Algorithms and complexity

Algorithms Algorithms Complexity and Data Complexity and Data Structures EfficiencyStructures Efficiency

Computational Complexity, Choosing Computational Complexity, Choosing Data StructuresData Structures

Svetlin NakovSvetlin NakovTelerik Telerik

CorporationCorporationwww.telerik.com

Table of ContentsTable of Contents

1.1. Algorithms Complexity and Algorithms Complexity and Asymptotic NotationAsymptotic Notation Time and Memory ComplexityTime and Memory Complexity Mean, Average and Worst CaseMean, Average and Worst Case

2.2. Fundamental Data Structures – Fundamental Data Structures – ComparisonComparison Arrays vs. Lists vs. Trees vs. Hash-Arrays vs. Lists vs. Trees vs. Hash-

TablesTables

3.3. Choosing Proper Data StructureChoosing Proper Data Structure

2

Why Data Structures Why Data Structures are Important?are Important?

Data structuresData structures and and algorithmsalgorithms are the are the foundation of computer programmingfoundation of computer programming

Algorithmic thinking, problem solving Algorithmic thinking, problem solving and data structures are vital for and data structures are vital for software engineerssoftware engineers All .NET developers should know when to All .NET developers should know when to

use use T[]T[], , LinkedList<T>LinkedList<T>, , List<T>List<T>, , Stack<T>Stack<T>, , Queue<T>Queue<T>, , Dictionary<K,T>Dictionary<K,T>, , HashSet<T>HashSet<T>, , SortedDictionary<K,T>SortedDictionary<K,T> and and SortedSet<T>SortedSet<T>

Computational complexity is important Computational complexity is important for algorithm design and efficient for algorithm design and efficient programmingprogramming

3

Algorithms Algorithms ComplexityComplexityAsymtotic NotationAsymtotic Notation

Algorithm AnalysisAlgorithm Analysis Why we should analyze algorithms?Why we should analyze algorithms?

Predict the resources that the Predict the resources that the algorithm requiresalgorithm requires Computational time (CPU consumption)Computational time (CPU consumption)

Memory space (RAM consumption)Memory space (RAM consumption)

Communication bandwidth consumptionCommunication bandwidth consumption

The The running timerunning time of an algorithm is: of an algorithm is: The total number of primitive operations The total number of primitive operations

executed (machine independent steps)executed (machine independent steps)

Also known as Also known as algorithm complexityalgorithm complexity

5

Algorithmic ComplexityAlgorithmic Complexity What to measure?What to measure?

MemoryMemory

TimeTime

Number of stepsNumber of steps

Number of particular operationsNumber of particular operations

Number of disk operationsNumber of disk operations

Number of network packetsNumber of network packets

Asymptotic complexityAsymptotic complexity

6

Time ComplexityTime Complexity Worst-caseWorst-case

An upper bound on the running time An upper bound on the running time for any input of given sizefor any input of given size

Average-caseAverage-case Assume all inputs of a given size Assume all inputs of a given size

are equally likelyare equally likely Best-caseBest-case

The lower bound on the running The lower bound on the running timetime

7

Time Complexity – Time Complexity – ExampleExample

Sequential search in a list of size nSequential search in a list of size n Worst-case:Worst-case:

nn comparisons comparisons

Best-case:Best-case: 11 comparison comparison

Average-case:Average-case: n/2n/2 comparisons comparisons

The algorithm runs in The algorithm runs in linear timelinear time Linear number of operationsLinear number of operations

…… …… …… …… …… …… ……

nn

8

Algorithms ComplexityAlgorithms Complexity Algorithm complexityAlgorithm complexity is rough estimation of is rough estimation of

the number of steps performed by given the number of steps performed by given computation depending on the size of the computation depending on the size of the input datainput data

Measured through asymptotic notationMeasured through asymptotic notation

O(g)O(g) where where gg is a function of the input data size is a function of the input data size

Examples:Examples:

Linear complexity Linear complexity O(n)O(n) – all elements are – all elements are processed once (or constant number of processed once (or constant number of times)times)

Quadratic complexity Quadratic complexity O(nO(n22)) – each of the – each of the elements is processed elements is processed nn times times

9

Asymptotic Notation: Asymptotic Notation: DefinitionDefinition

Asymptotic upper boundAsymptotic upper bound O-notation (Big O notation)O-notation (Big O notation)

For given function For given function g(n)g(n), we denote by , we denote by O(g(n))O(g(n)) the set of functions that are the set of functions that are different than different than g(n)g(n) by a constant by a constant

Examples:Examples: 33 ** nn22 ++ n/2n/2 ++ 12 12 ∈∈ O(nO(n22)) 4*n*log4*n*log22(3*n+1)(3*n+1) ++ 2*n-1 2*n-1 ∈∈ O(nO(n ** loglog n)n)

O(g(n))O(g(n)) == { {f(n)f(n): there exist positive : there exist positive constants constants cc and and nn00 such that such that f(n)f(n) <=<= c*g(n)c*g(n) for all for all nn >=>= nn00}}

10

Typical ComplexitiesTypical Complexities

11

ComplexComplexityity

NotatiNotationon DescriptionDescription

constantconstant O(1)O(1)

Constant number of Constant number of operations, not operations, not depending on the depending on the input data size, e.g.input data size, e.g.n = 1 000 000 n = 1 000 000 1-2 1-2 operationsoperations

logarithlogarithmicmic

O(logO(log n)n)

Number of operations Number of operations propor-tional propor-tional of logof log22(n) (n) where n is the size of where n is the size of the input data, e.g. n the input data, e.g. n = 1 000 000 000 = 1 000 000 000 30 30 operationsoperations

linearlinear O(n)O(n)

Number of operations Number of operations proportional to the proportional to the input data size, e.g. n input data size, e.g. n = 10 000 = 10 000 5 000 5 000 operationsoperations

Typical Complexities (2)Typical Complexities (2)

12

ComplexiComplexityty

NotatiNotationon DescriptionDescription

quadraticquadratic O(nO(n22))

Number of operations Number of operations proportional to the proportional to the square of the size of square of the size of the input data, e.g. n the input data, e.g. n = 500 = 500 250 000 250 000 operationsoperations

cubiccubic O(nO(n33))

Number of operations Number of operations propor-tionalpropor-tional to the to the cube of the size of the cube of the size of the input data, e.g. n =input data, e.g. n =200 200 8 000 000 8 000 000 operationsoperations

exponentexponentialial

O(2O(2nn)),,O(O(kknn)),,O(n!)O(n!)

Exponential number of Exponential number of operations, fast operations, fast growing, e.g. n = 20 growing, e.g. n = 20 1 048 576 operations1 048 576 operations

Time Complexity and Time Complexity and SpeedSpeed

13

ComplexComplexityity 1010 2020 5050 100100 11

0000001010 000000

100100 000000

O(1)O(1) < < 11 ss< < 11 ss

< < 11 ss < < 11 ss < < 11 ss < < 11 ss < < 11 ss

O(log(n))O(log(n)) < < 11 ss< < 11 ss

< < 11 ss < < 11 ss < < 11 ss < < 11 ss < < 11 ss

O(n)O(n) < < 11 ss< < 11 ss

< < 11 ss < < 11 ss < < 11 ss < < 11 ss < < 11 ss

O(n*log(nO(n*log(n))))

< < 11 ss< < 11 ss

< < 11 ss < < 11 ss < < 11 ss < < 11 ss < < 11 ss

O(nO(n22)) < < 11 ss< < 11 ss

< < 11 ss < < 11 ss < < 11 ss 22 ss33--44 minmin

O(nO(n33)) < < 11 ss< < 11 ss

< < 11 ss < < 11 ss 2020 s s55

hourshours231231

daysdays

O(2O(2nn)) < < 11 ss< < 11 ss

260260 daysdays

hanhangsgs

hanhangsgs

hanghangss

hangshangs

O(n!)O(n!) < < 11 sshanhangsgs

hanhangsgs

hanhangsgs

hanhangsgs

hanghangss

hangshangs

O(nO(nnn))33--44 minmin

hanhangsgs

hanhangsgs

hanhangsgs

hanhangsgs

hanghangss

hangshangs

Time and Memory Time and Memory ComplexityComplexity

Complexity can be expressed as formula Complexity can be expressed as formula on multiple variables, e.g.on multiple variables, e.g.

Algorithm filling a matrix of size Algorithm filling a matrix of size nn * * mm with with natural numbers natural numbers 11, , 22, … will run in , … will run in O(n*m)O(n*m)

DFS traversal of graph with DFS traversal of graph with nn vertices vertices and and mm edges will run in edges will run in O(nO(n ++ m)m)

Memory consumption should also be Memory consumption should also be considered, for example:considered, for example:

Running time Running time O(n)O(n), memory requirement , memory requirement O(nO(n22))

n = 50 000 n = 50 000 OutOfMemoryExceptionOutOfMemoryException14

Polynomial AlgorithmsPolynomial Algorithms A A polynomial-time algorithmpolynomial-time algorithm is one is one

whose worst-case time complexity is whose worst-case time complexity is bounded above by a polynomial function bounded above by a polynomial function of its input sizeof its input size

Example of worst-case time complexityExample of worst-case time complexity Polynomial-time:Polynomial-time: log log nn, , 2n2n, , 3n3n33 ++ 4n4n, , 22 * * nn

log log nn Non polynomial-time : Non polynomial-time : 22nn, , 33nn,, nnkk, , n!n!

Non-polynomial algorithms don't work Non-polynomial algorithms don't work for large input data setsfor large input data sets

W(n) W(n) ∈ O(p(n)) O(p(n))

15

Analyzing Analyzing Complexity of Complexity of

AlgorithmsAlgorithmsExamplesExamples

Complexity ExamplesComplexity Examples

Runs in Runs in O(n)O(n) where where nn is the size of the is the size of the arrayarray

The number of elementary steps is The number of elementary steps is ~~ nn

int FindMaxElement(int[] array)int FindMaxElement(int[] array){{ int max = array[0];int max = array[0]; for (int i=0; i<array.length; i++)for (int i=0; i<array.length; i++) {{ if (array[i] > max)if (array[i] > max) {{ max = array[i];max = array[i]; }} }} return max;return max;}}

Complexity Examples Complexity Examples (2)(2)

Runs in Runs in O(nO(n22)) where where nn is the size of is the size of the arraythe array

The number of elementary steps is The number of elementary steps is ~~ n*(n+1)n*(n+1) // 22

long FindInversions(int[] array)long FindInversions(int[] array){{ long inversions = 0;long inversions = 0; for (int i=0; i<array.Length; i++)for (int i=0; i<array.Length; i++) for (int j = i+1; j<array.Length; i+for (int j = i+1; j<array.Length; i++)+) if (array[i] > array[j])if (array[i] > array[j]) inversions++;inversions++; return inversions;return inversions;}}


Runs in cubic time Runs in cubic time O(nO(n33)) The number of elementary steps is The number of elementary steps is ~~ nn33

decimal Sum3(int n)decimal Sum3(int n){{ decimal sum = 0;decimal sum = 0; for (int a=0; a<n; a++)for (int a=0; a<n; a++) for (int b=0; b<n; b++)for (int b=0; b<n; b++) for (int c=0; c<n; c++)for (int c=0; c<n; c++) sum += a*b*c;sum += a*b*c; return sum;return sum;}}


Runs in quadratic time Runs in quadratic time O(n*m)O(n*m) The number of elementary steps is The number of elementary steps is ~~ n*mn*m

long SumMN(int n, int m)long SumMN(int n, int m){{ long sum = 0;long sum = 0; for (int x=0; x<n; x++)for (int x=0; x<n; x++) for (int y=0; y<m; y++)for (int y=0; y<m; y++) sum += x*y;sum += x*y; return sum;return sum;}}


Runs in quadratic time Runs in quadratic time O(n*m)O(n*m) The number of elementary steps is The number of elementary steps is

~~ n*mn*m ++ min(m,n)*nmin(m,n)*n

long SumMN(int n, int m)long SumMN(int n, int m){{ long sum = 0;long sum = 0; for (int x=0; x<n; x++)for (int x=0; x<n; x++) for (int y=0; y<m; y++)for (int y=0; y<m; y++) if (x==y)if (x==y) for (int i=0; i<n; i++)for (int i=0; i<n; i++) sum += i*x*y;sum += i*x*y; return sum;return sum;}}


Runs in exponential time Runs in exponential time O(2O(2nn)) The number of elementary steps is The number of elementary steps is ~~ 22nn

decimal Calculation(int n)decimal Calculation(int n){{ decimal result = 0;decimal result = 0; for (int i = 0; i < (1<<n); i++)for (int i = 0; i < (1<<n); i++) result += i;result += i; return result;return result;}}


Runs in linear time Runs in linear time O(n)O(n) The number of elementary steps is The number of elementary steps is ~~ nn

decimal Factorial(int n)decimal Factorial(int n){{ if (n==0)if (n==0) return 1;return 1; elseelse return n * Factorial(n-1);return n * Factorial(n-1);}}


Runs in Runs in exponential time exponential time O(2O(2nn)) The number of elementary steps is The number of elementary steps is

~~ Fib(n+1)Fib(n+1) w where here Fib(k)Fib(k) is the is the kk-th Fib-th Fiboonacci's numbernacci's number

decimal Fibonacci(int n)decimal Fibonacci(int n){{ if (n == 0)if (n == 0) return 1;return 1; else if (n == 1)else if (n == 1) return 1;return 1; elseelse return Fibonacci(n-1) + Fibonacci(n-2);return Fibonacci(n-1) + Fibonacci(n-2);}}

Comparing Data Comparing Data StructuresStructures

ExamplesExamples

Data Structures Data Structures EfficiencyEfficiency

26

Data Data StructureStructure AddAdd FinFin

ddDeletDelet

ee

Get-Get-by-by-

indexindexArray (Array (T[]T[])) O(n)O(n) O(n)O(n) O(n)O(n) O(1)O(1)

Linked list Linked list ((LinkedList<T>LinkedList<T>

))O(1)O(1) O(n)O(n) O(n)O(n) O(n)O(n)

Resizable Resizable array list array list ((List<T>List<T>))

O(1)O(1) O(n)O(n) O(n)O(n) O(1)O(1)

Stack Stack ((Stack<T>Stack<T>)) O(1)O(1) -- O(1)O(1) --

Queue Queue ((Queue<T>Queue<T>)) O(1)O(1) -- O(1)O(1) --

Data Structures Data Structures EfficiencyEfficiency (2) (2)

27

Data Data StructureStructure AddAdd FindFind DeletDelet

ee

Get-Get-by-by-

indexindexHash table Hash table

((Dictionary<K,TDictionary<K,T>>))

O(1)O(1) O(1)O(1) O(1)O(1) --

Tree-based Tree-based dictionary dictionary

((Sorted Sorted Dictionary<K,TDictionary<K,T

>>))

O(logO(log n)n)

O(logO(log n)n)

O(logO(log n)n) --

Hash table Hash table based set based set

((HashSet<T>HashSet<T>))O(1)O(1) O(1)O(1) O(1)O(1) --

Tree based set Tree based set ((SortedSet<T>SortedSet<T>))

O(logO(log n)n)

O(logO(log n)n)

O(logO(log n)n) --

Choosing Data Choosing Data StructureStructure

Arrays (Arrays (T[]T[])) Use when fixed number of elements Use when fixed number of elements

should be processed by indexshould be processed by index Resizable array lists (Resizable array lists (List<T>List<T>))

Use when elements should be added Use when elements should be added and processed by indexand processed by index

Linked lists (Linked lists (LinkedList<T>LinkedList<T>)) Use when elements should be added Use when elements should be added

at the both sides of the listat the both sides of the list Otherwise use resizable array list Otherwise use resizable array list

((List<T>List<T>)) 28

Choosing Data Choosing Data Structure (2)Structure (2)

Stacks (Stacks (Stack<T>Stack<T>)) Use to implement LIFO (last-in-first-out) Use to implement LIFO (last-in-first-out)

behaviorbehavior List<T>List<T> could also work well could also work well

Queues (Queues (Queue<T>Queue<T>)) Use to implement FIFO (first-in-first-out) Use to implement FIFO (first-in-first-out)

behaviorbehavior LinkedList<T>LinkedList<T> could also work well could also work well

Hash table based dictionary (Hash table based dictionary (Dictionary<K,T>Dictionary<K,T>)) Use when key-value pairs should be added fast Use when key-value pairs should be added fast

and searched fast by keyand searched fast by key Elements in a hash table have no particular Elements in a hash table have no particular

orderorder29

Choosing Data Choosing Data Structure (3)Structure (3)

Balanced search tree based dictionary Balanced search tree based dictionary ((SortedDictionary<K,T>SortedDictionary<K,T>)) Use when key-value pairs should be added Use when key-value pairs should be added

fast, searched fast by key and enumerated fast, searched fast by key and enumerated sorted by keysorted by key

Hash table based set (Hash table based set (HashSet<T>HashSet<T>)) Use to keep a group of unique values, to Use to keep a group of unique values, to

add and check belonging to the set fastadd and check belonging to the set fast Elements are in no particular orderElements are in no particular order

Search tree based set (Search tree based set (SortedSet<T>SortedSet<T>)) Use to keep a group of ordered unique Use to keep a group of ordered unique

valuesvalues30

SummarySummary Algorithm complexityAlgorithm complexity is rough estimation is rough estimation

of the number of steps performed by of the number of steps performed by given computationgiven computation

Complexity can be logarithmic, linear, n log Complexity can be logarithmic, linear, n log n, square, cubic, exponential, etc.n, square, cubic, exponential, etc.

Allows to estimating the speed of given Allows to estimating the speed of given code before its executioncode before its execution

Different data structures have different Different data structures have different efficiency on different operationsefficiency on different operations The fastest add / find / delete structure The fastest add / find / delete structure

is the hash table – is the hash table – O(1)O(1) for all these for all these operationsoperations

31

Algorithms Complexity Algorithms Complexity and Data Structures and Data Structures

EfficiencyEfficiency

Questions?Questions?

http://academy.telerik.com

ExercisesExercises

1.1. A text file A text file students.txtstudents.txt holds holds information about students and their information about students and their courses in the following format:courses in the following format:

Using Using SortedDictionary<K,T>SortedDictionary<K,T> print the print the courses in alphabetical order and for courses in alphabetical order and for each of them prints the students each of them prints the students ordered by family and then by name:ordered by family and then by name:

33

Kiril | Ivanov | C#Kiril | Ivanov | C#Stefka | Nikolova | SQLStefka | Nikolova | SQLStela | Mineva | JavaStela | Mineva | JavaMilena | Petrova | C#Milena | Petrova | C#Ivan | Grigorov | C#Ivan | Grigorov | C#Ivan | Kolev | SQLIvan | Kolev | SQL

C#: Ivan Grigorov, Kiril Ivanov, Milena PetrovaC#: Ivan Grigorov, Kiril Ivanov, Milena PetrovaJava: Stela MinevaJava: Stela MinevaSQL: Ivan Kolev, Stefka NikolovaSQL: Ivan Kolev, Stefka Nikolova

Exercises (2)Exercises (2)2.2. A large trade company has millions of articles, A large trade company has millions of articles,

each described by barcode, vendor, title and each described by barcode, vendor, title and price. Implement a data structure to store them price. Implement a data structure to store them that allows fast retrieval of all articles in given that allows fast retrieval of all articles in given price range price range [x…y][x…y]. Hint: use . Hint: use OrderedMultiDictionary<K,T>OrderedMultiDictionary<K,T> from from Wintellect's Power Collections for .NET.

3.3. Implement a data structure Implement a data structure PriorityQueue<T>PriorityQueue<T> that provides a fast way to execute the that provides a fast way to execute the following operations:following operations: add element; extract the add element; extract the smallest element.smallest element.

4.4. Implement a class Implement a class BiDictionary<K1,K2,T>BiDictionary<K1,K2,T> that that allows adding triples allows adding triples {key1,{key1, key2,key2, value}value} and and fast search by fast search by key1key1, , key2key2 or by both or by both key1key1 and and key2key2. Note: multiple values can be stored for . Note: multiple values can be stored for given key.given key.

34

Exercises (3)Exercises (3)5.5. A text file A text file phones.txtphones.txt holds information holds information

about people, their town and phone about people, their town and phone number:number:

Duplicates can occur in people names, Duplicates can occur in people names, towns and phone numbers. Write a towns and phone numbers. Write a program to execute a sequence of program to execute a sequence of commands from a file commands from a file commands.txtcommands.txt:: find(name)find(name) – display all matching records by – display all matching records by

given name (first, middle, last or nickname)given name (first, middle, last or nickname)

find(name,find(name, town)town) – display all matching – display all matching records by given name and townrecords by given name and town 35

Mimi Shmatkata | Plovdiv | 0888 12 34 56Mimi Shmatkata | Plovdiv | 0888 12 34 56Kireto | Varna | 052 23 45 67Kireto | Varna | 052 23 45 67Daniela Ivanova Petrova | Karnobat | 0899 999 888Daniela Ivanova Petrova | Karnobat | 0899 999 888Bat Gancho | Sofia | 02 946 946 946Bat Gancho | Sofia | 02 946 946 946

Technology

19 Algorithms and complexity