Unit 5 Decision Trees

8/17/2019 Unit 5 Decision Trees

1/56

1

Lower Bounds

Lower bound : an estimate on aminimum amount of work needed tosolve a given problem

Lower bound can bean exact count

an eciency class (

)

Tight lower bound: there exists analgorithm with the same eciency as

the lower bound


2/56

2

ethods for !stablishing

Lower Bounds

• trivial lower bounds

• information-theoretic arguments(decision trees)

• adversary arguments

• problem reduction


3/56

3

Trivial Lower Bounds

• Based on counting the number ofitems that must be processed in

input and generated as output

• !xamples

"nding max element

sorting

element uni#ueness$ot always

useful


4/56

4

%ecision Trees

& convenient model of algorithms

involving comparisons in which:

• internal nodesinternal nodes represent

comparisons• leavesleaves represent outcomes


5/56


6/56

6

&dversary &rguments

&dversary argument: a method ofproving a lower bound by playing role

of adversary that makes algorithmwork the hardest by ad'usting input

!xample: uessing* a number

between + and n with yes,no#uestions

&dversary: -uts the number in alarger of the two subsets generated


7/56

7

Lower Bounds by -roblem.eduction

/dea: /f problem P is at least ashard as problem Q0 then a lower

bound for Q is also a lower boundfor P.

1ence0 "nd problem Q with aknown lower bound that can bereduced to problem P in #uestion2

Then any algorithm that solves -


8/56

Class PP: the class of decision problems that are solvable in

O( p(n)) time, here p(n) is a pol!nomial of problem"sinp#t si$e n

Examples:

• searching

• element uniqueness

• graph connectivity

• graph acyclicity


9/56

Class NP

NP (nondeterministic polynomial): class of decisionproblems hose proposed sol#tions can be veri%ed inpol!nomial time & solvable b! a nondeterministic

polynomial algorithm

' nondeterministic polynomial algorithm is an abstracttostae proced#re that:

• enerates a sol#tion of the problem (on some inp#t)b! #essin

•chec*s hether this sol#tion is correct in pol!nomialtime

+! de%nition, it solves the problem if it"s capable ofeneratin and verif!in a sol#tion on one of its tries


10/56

-ample: C./ satis%abilit!0roblem: s a boolean e-pression in its con#nctive normal

form (C./) satis%able, ie, are there val#es of its variablesthat ma*e it tr#e

5his problem is in NP .ondeterministic alorithm:• #ess a tr#th assinment

• #bstit#te the val#es into the C./ form#la to see if iteval#ates to tr#e

-ample: (' 8 9+ 8 9C) (' 8 +) (9+ 8 9; 8 ) (9; 8 9) 5r#th assinments:

' + C ;

< < < < <

1 1 1 1 1

Chec*in phase: O(n)


11/56


12/56

NPComplete 0roblems

' decision problem D is NPcomplete if it is as hard asan!

problem in NP, ie,

• D is in NP

• ever! problem in NP is pol!nomialtime red#cible to D

Coo*"s theorem (1=71): C./sat is NPcomplete

NP -complete

problem

NP problems


13/56

13

Backtracking

• Suppose you have to make a series ofdecisions , among various choices , where

– You don’t have enough information to know

what to choose – Each decision leads to a new set of choices

– Some sequence of choices (possibly more than

one) may be a solution to your problem

• Backtracking is a methodical way of trying

out various sequences of decisions until you

find one that !works"


14/56

+ac*trac*in

• #onstruct the state-space tree $State space is the set of allpaths from root node to other nodes %

– nodes& partial solutions

– edges& choices in e'tending partial solutions

• E'plore the state space tree using depthfirst search

• !rune" nonpromising nodes – *+S stops e'ploring subtrees rooted at nodes that cannot

lead to a solution and backtracks to such a node’s parent to

continue the search


15/56

1>

Backtracking Algorithm

• Based on depth-frst recursive search

• Approach

1. Tests whether solution has een !ound

". #! !ound solution$ return it

%. Else !or each choice that can e made

a& 'a(e that choice

& )ecur

c& #! recursion returns a solution$ return it

*. #! no choices remain$ return !ailure

• +ome times called ,search tree


16/56

Example: n -Queens Problem

Place n queens on an n- by-n chess board sothat no two of them are in the same rowcolumn or diagonal

1 2 3 4

1

2

34

queen 1

queen 2

queen 3queen 4


17/56

!tate-!pace "ree of the #-Queens Problem


18/56

Hamiltonian Circuits Problem

A Hamiltonian circuit or tour of a graph is apath that starts at a given vertex, visits eachvertex in the graph exactly once, and ends atthe starting vertex. Some graphs do notcontain Hamiltonian circuits.v 1 v 2

v 6 v 4 v 5

v 3

A state space tree for this problem is as follows. Put the starting vertex at level

in the tree! call this the "ero#th vertex on the path. At level 1! consi$er each

vertex other than the starting vertex as the first vertex after the starting one. Atlevel 2! consi$er each of these vertices as the secon$ vertex! an$ so on. %ou

ma& now bac'trac' in this state space tree.

v 1 v 2

v 6 v 4 v 5

v 3


19/56


20/56

State-space tree for nding a Hamiltonian circuit. Thenumbers above the nodes of the tree indicate the order in

which the nodes are generated.


21/56

!ubset-sum problem

– iven positive numers wi$ 1/i/n$ and

m$ fnd all susets o! the wi whose sum

is m

• eg) (w 1,w ,w !,w ") # (11,1!,",$) andm#!1 – Solutions are (11,1!,$) and (",$)


22/56

$ethod

• Sort the set’s elements in increasing order

• ,he root of the tree represents the startingpoint with no decisions about the given elements made as yet-

• .ts left and right children represent,

respectivel%, inclusion and e&clusion of (rstelement) in a set being sought.

• Similarly going to the left from a node of the first level

corresponds to inclusion of (second element) while going to the

right corresponds to its exclusion, and so on.• Thus, a path from the root to a node on the i th level of the tree

indicates which of the first I numbers have been included in the

subsets represented by that node-


23/56

• /e record the value of s, the sum of these numbers ,

in the node.

– If s is equal to d, we have a solution to the problem.

– We can either report this result and stop

• or if all the solutions need to be found continue by

backtracking to the node’s parent-

• /f s is not equal to d , we can terminatethe node as nonpromising if either of the

folloin to ine?#alities holds:


24/56


25/56

0

0

05

11 5

3

38

3

with 3

with 5

with 6

w/o 3

w/o 5

w/o 6 with 6 w/o 6

w/o 5 with 5

X X X X

X

14+7>15 3+714 5+7


26/56

Branch and Bound


27/56

The branch-and-bound problem solving method is verysimilar to ac(trac(ing in that a state space tree is used tosolve a prolem. The dierences are that B2B

(1)does not limit us to an% particular wa% of traversingthe tree and

3"& is used onl% for optimi'ation problems.

' ++ alorithm comp#tes a n#mber (bo#nd) at a node to determinehether the node is promisin 5he n#mber is a bo#nd on the val#e ofthe sol#tion that co#ld be obtained b! e-pandin the state space treebe!ond the c#rrent node


28/56

2A

"he main idea

• !et up a bounding function which is used to compute a bound%for the &alue of the ob'ecti&e function( at a node on a state-space tree and determine if it is promising

Promising %if the bound is better than the &alue of thebest soluton so far:expand beyond the node)

Nonpromising %if the bound is no better than the &alue ofthe best solution so far--:not smaller for a minimi*ationproblem and not larger for a maxmi*ation problem (: notexpand beyond the node %pruning the state-space tree()


29/56

2=

Terminate a search path at the current node in a state-space tree forany one of the following three reasons:

"he &alue of the node+s bound is not better than the &alue of

the best solution seen so far)

"he node reprsents no feasible solutions because theconstraints of the problem are already &iolated)

"he subset of feasible solutions reprsented by the nodeconsists of a single point%and hence no further choices can bemade(-in this case we compare the &alue of the ob'ecti&efunction for this feasible solution with that of the bestsolution seen so far and update the latter with the former ifthe new solution is better)


30/56

3<

Assignment problem

Assigning n people to n 'obs so that the total cost of theassignment is as small as possible


31/56

31

Start

Ib10

a ￫ 1

Ib1!

a ￫ 2

Ib10

a ￫ 3

Ib20

a ￫ 4

Ib18

b ￫ 1

Ib13

b ￫ 3

Ib14

b ￫ 4

Ib1!

c￫

3Ib13

c￫

4lb25

" ￫ 4

#ost13

0

1 2 3 4

5 $ !

8 9

10complete state-space tree for the

instance of the assignment problem

solved with the best-first branch-and-

bound algorithm


32/56

32

,napsack problem

• iven n items of *non eihts wi and values vi ,

• .t is convenient to order the items of a given instance in descending order

by their value-to-weight ratios- ,hen the first item gives the best payoff

per weight unit and the last one gives the worst payoff per weight unit

with ties resolved arbitrarily

31& v14w15 v"4w" 56 5 vn4wn

3"& ound !unction: u7v839-w&3vi814wi81&


33/56

33

-ample

Item %e&'t alue alue*%e&'t

1 4 40 10

2 ! 42 $

3 5 25 5

4 3 12 4

+10


34/56

34

%0,0

-b100

%4,40

-b!$

%0,0

-b$0

%11 %4,40

-b!0

%9,$5

-b$9%4,40

-b$4

%12 %9,$5

-b$5

+&t 1 +*o 1


35/56


36/56

36

5ravelin alesman 0roblemB+o#ndin/#nction

• +eca#se ever! verte- m#st be entered and e-ited e-actl! once, aloer bo#nd on the lenth of a to#r is the s#m of the minim#mcost of enterin and leavin ever! verte- – 4or a given edge (u0 v)0 think of half of its weight as the the

exiting cost of u0 and half of its weight as the entering cost of

v2 – The total length of a tour 5 the total cost of visiting( entering

and exiting) every vertex exactly once2 – The lower bound of the length of a tour 5 the lower bound of

the total cost of visiting (entering and exiting ) every vertexexactly once2

• 6alculation: – for each vertex0 pick top two shortest ad'acent edges (their sum

divided by 7 is the lower bound of the total cost of entering andexiting the vertex)8

– add up these summations for all the vertices2

• 'ss#me that the to#r starts ith verte- a and that b is visitedbefore c


37/56

37

5ravelin salesman

a

Ib14

a,bIb14

a,c a,"Ib1$

a,e

Ib19

0

1 23 4

5 $ !

9

a,b,c

Ib1$

a,b,e

Ib19

a,b,"

Ib1$

8

a,b,c,",e,a

l24

10 11

b &s not be.ore c

a,b,c,",e,a

l19

a,b,",c,e,a

I24

a,b,",e,c,a

I=16

' i ti ' h


38/56

'ppro-imation 'pproach

Apply a !ast 3i.e.$ a polynomial-time& approximation

algorithm to get a solution that is not necessarilyoptimal ut hope!ully close to it

Accuracy measures:

accuracy ratio o! an approximate solution sa

r 3sa& 7 !3sa& 4 !3s& !or minimi;ation prolems

r 3sa& 7 !3s& 4 !3sa& !or maximi;ation prolems

where !3sa& and !3s& are values o! the o


39/56

• A polynomial-time approximationalgorithm is said to e a capproximation algorithm, where c

≥ , if the accuracy ratio of theapproximation

• it produces does not exceed c for any

instance of the problem in question: r(sa) ≤ c


40/56

• A heuristic is a common-sense ruledrawn !rom experience rather than!rom a mathematically proved

assertion.

• =or example$ going to the nearestunvisited city in the traveling

salesman prolem is a goodillustration o! this notion.

Approximation Algorithms !or the


41/56

Approximation Algorithms !or theTraveling

+alesman >rolem

0iven n cities with known distances between

each pair find the shortest tour that passes

through all the cities e'actly once before

returning to the starting city

1lternatively& +ind shortest Hamiltonian

circuit in a weighted connected graph


42/56

-ample:

a b

c d

8

2

!

5 34

50 b h ti h


43/56

50 b! -ha#stive earch

,our #ost

a2b2c2d2a 3454647 8 96

a2b2d2c2a 34:464; 8 39

a2c2b2d2a ;454:47 8 3<

a2c2d2b2a ;464:43 8 39a2d2b2c2a 74:454; 8 3<

a2d2c2b2a 7464543 8 96

"h t i hb


44/56

"he earest eighbourAlgorithm

• ,he following wellknown greedy algorithm is based on the

nearest-neighbor

• heuristic& always go ne't to the nearest unvisited city-

• Step 1 Choose an arbitrary city as the start.

• Step 2 Repeat the following operation until all the cities

have been visited:

• go to the unvisited city nearest the one visited last (ties can be

broken arbitrarily)-

• Step 3 Return to the starting city.


45/56

u t ragment- eur st c


46/56

u t ragment- eur st c Algorithm

+tage 1: +ort the edges in nondecreasing order o! weights.

#nitiali;e the set o! tour edges to e an empty set

+tage ": Add next edge on the sorted list to the tour$ s(ippingthose whose addition would have created a vertex o! degree % or a cycle o! length less than n. )epeat

this step until a tour o! length n is otainedtep 3 et#rn the set of to#r edes


47/56

• D(a, b), (c, d), (b, c), (a, d)}

'i i + i T


48/56

'inimum-+panning-Tree@Based Algorithms

ce ro#n e ree


49/56

ce ro#n e ree'lorithm

Stage 9& #onstruct a minimum spanning tree of the graph

(e-g- by rim’s or =ruskal’s algorithm)

Stage 3& Starting at an arbitrary verte' create a path that goes

twice around the tree and returns to the same verte'

Stage 5& #reate a tour from the circuit constructed in Stage 3 by

making shortcuts to avoid visiting intermediate vertices

more than once


50/56


51/56

Eocal earch Fe#ristics for


52/56

Eocal earch Fe#ristics for 50

tart ith some initial to#r (e, nearest neihbor) Oneach iteration, e-plore the c#rrent to#r/s neihborhoodb! e-chanin a fe edes in it f the ne to#r isshorter, ma*e it the c#rrent to#rG otherise consideranother ede chane f no chane !ields a shorter

to#r, the c#rrent to#r is ret#rned as the o#tp#t

-ample of a 2chane

#1

#2

#4

#3

#1 #2

#4 #3


53/56

>3

2opt

• 5a*e to edes(v ,w) and ( x,y ) andreplace them b!

(v,x ) and (w,y ) O(v,y ) and (w,x ) toet a to#r aain

• Costl!: part of to#r

sho#ld be t#rnedaro#nd


54/56


55/56

Hnapsac* problem:

/nput: n items: (1, v1), I, (n, vn) and an#mber J

– the ith item"s is orth vi dollars ith eiht i

– at most J po#nds to be carried

9utput: ome items hich are mostval#able and ith total eiht at most J

• 5o versions: – ;+

Solve the fractional napsac!


56/56

Solve the fractional napsac!

problem: • "reedy on the value per pound v i #w i $

– Each time take the item with maximum & i .w i )

– /f exceeds 0 take fractions of the item) • Example: %1 23( %4 53( %6 143( %6 113( and w7#)• &i.wi : 2 #)2 #)8 6)99• ;irst: %1 23( !econd: %4 53( "hird: 1.6 %6

143(• "otal 0: 1 6 #)

Documents

Unit 5 Decision Trees