Hybrid of search and inference: time-space tradeoffs chapter 10

1

Hybrid of search and inference: time-Hybrid of search and inference: time-space tradeoffsspace tradeoffs

chapter 10chapter 10

ICS-275

Spring 2007

Spring 2007 ICS-275 3

Chapter 10:Hybrids of Search and Inference

Interleaving conditioning and elimination

Cutset decomposition AND/OR cutset decomposition Super-bucket and super-clusters Approximating conditioning in a

hybrid


Satisfiability: Inference vs search

))exp(( :space and timeDR))(exp(||

*

*

wnOwObucketi

Search = O(exp(n))Search = exp(dfs-height)

)()()()( CDCBEBACBA


DR versus DPLL: Complementary Properties

Uniform random 3-CNFs(large induced width)

(k,m)-tree 3-CNFs(bounded induced width)


Outline; Road Map

CSP SAT OptimizationBelief

updatingMPE, MAP,

MEU

Solving Linear Equalites / Inequalities

eliminationadaptive

consistency join-tree

directional resolution

dynamic programing

join-tree, VE, SPI, elim-bel

join-tree, elim-mpe, elim-map

Gaussian / Fourier

elimination

conditioningbacktracking

search

backtracking (Davis-Putnam)

branch-and-bound, best-first search

branch-and-bound, best-first search

elimination + conditioning

cycle-cutset, forward

checking

DCDR, BDR-DP

loop-cutset

approximate elimination

i-consistencybounded

(directional) resolution

mini-bucketsmini-

bucketsmini-buckets

approximate conditioning

greedylocal search (GSAT)

GSATgradient descent

stochastic simulation

gradient descent

approximate (elimination + conditioning)

GSAT + partial path-consistency

MethodsTasks


The effect of Conditioning

X1

X3

X5X4

X2



X1

X3

X5X4

X2

•Select a variable



X1

X3

X5X4

X2

X3

X5X4

X2

X3

X5X4

X2

X3

X5X4

X2

…...

…...

X1 aX1 b

X1 c



X1

X3

X5X4

X2

X3

X5X4

X2

X3

X5X4

X2

X3

X5X4

X2

…...

…...

X1 aX1 b

X1 c

Condition on X2, andBE or BE(w=2) now.



X1

X3

X5X4

X2

X3

X5X4

X2

X3

X5X4

X2

X3

X5X4

X2

…...

…...

X1 aX1 b

X1 c

General principle:Condition until tractable


Interleaving Cond and Elim (VE+C)


Interleaving Cond and Elim











...

...



...

...



...

...



...

...



...

...



...

...



...

...



...

...



...

...


otherwise condition

,)(w* bxi if Resolve


DCDR(b): empirical results

)exp(space ),_exp( Time

hybrid :0 DR,pure : DPLL,pure : 0

:off- tradeAdjustable **

bbcb

wbwbb


Road Map


Cutset decomposition• W-cutset, cycle-cutset

• AND/OR w-cutset

Super-bucket and super-clusters


Cycle cutset

C P

J A

L

B

E

DF M

O

H

K

G N

C P

J

L

B

E

DF M

O

H

K

G N

A

C P

J

L

E

DF M

O

H

K

G N

B

P

J

L

E

DF M

O

H

K

G N

C

Cycle cutset = {A,B,C}

C P

J A

L

B

E

DF M

O

H

K

G N

C P

J

L

B

E

DF M

O

H

K

G N

C P

J

L

E

DF M

O

H

K

G N

C P

J A

L

B

E

DF M

O

H

K

G N


Conditioning and cycle-cutset

A=yellow A=green

• Inference may require too much memory

• Condition (guessing) on some of the variables

C

B K

G

L

D

FH

M

J

E

C

B K

G

L

D

FH

M

J

E

AC

B K

G

L

D

FH

M

J

E

GraphColoringproblem


Conditioning

A=yellow A=green

B=red B=blue B=red B=blueB=green B=yellow

C

K

G

L

D

FH

M

J

E

C

K

G

L

D

FH

M

J

E

C

K

G

L

D

FH

M

J

E

C

K

G

L

D

FH

M

J

E

C

K

G

L

D

FH

M

J

E

C

K

G

L

D

FH

M

J

E


• Condition on some of the variablesA

C

B K

G

L

D

FH

M

J

E



The cycle-cutset scheme: condition until a tree


Theorem: The w-cutset scheme yields space complexity

exp(w) and time complexity exp(w+c_w), where c_w is the size of the w-cutset.

As w decreases, c_w increases.

The cycle-cutset decomposition is linear space and

Has time complexity of

))(( )2( ckcnO

Time-space tradeoff


Elim-cond(w) or w-cutset scheme Idea: runs backtracking search on the w-cutset variables and

bucket-elimination on the remaining variables.

Input: A constraint network R = (X,D,C), Y a w-cutset, d an ordering that starts with Y whose adjusted induced-width, along d, is bounded by w, Z = X-Y.

Output: A consistent assignment, if there is one. 1. while {y} next partial solution of Y found by backtracking,

do• a) z solution found by adaptive-consistency(R_y).• B) if z is not false, return solution (y,z).

2. endwhile. return: the problem has no solutions.


Finding a w-cutset

Verifying a w-cutset can be done in polynomial time

A simple greedy: use a good induced-width ordering and starting at the top add to the w-cutset any variable with more than w parents.

Alternative: generate a tree-decomposition and select a w-cutset that reduce each cluster below w (Bidyuk and Dechter UAI2005).


Treewidth equals cycle cutset

EK

F

H

C

BA

L

G

J

D

M

N

K

C

A

L

G

J

D

M

N

treewidth = cycle cutset = 4


Treewidth smaller than cycle cutset

EK

F

H

C

BA

L

G

J

D

F

C

B

GD

E

treewidth = 2

cycle cutset = 5


Time-Space complexity of of w-cutset

Space: O(exp(w))• W-cutset: a set that when removed the induced-

width is w.

• c(w): size of w-cutset.

Time: O(exp(w+c(w))) on OR space

,)(*

**)(*...)(...)2(21)1(

iictw

twwcwicicc


Time vs space

• Random Graphs (50 nodes, 200 edges, average degree 8, w*23)

Branch and bound

Bucket elimination

0

10

20

30

40

50

60

-1 2 5 8 11 14 17 20 23 26 29 32

w

W+

c(w

)


DCDR(b): empirical results(Rish and Dechter 2000)

)exp(space |),)(|exp( Time hybrid :0 DR,pure : DPLL,pure : 0

:off- tradeAdjustable **

bbcondbwbwbb


Empirical evaluation of w-cutset

Cycle-cutset for CSPs (dechter 1990) Alternating w-cutset for cnfs (Rish and

Dechter 2000) Alternating w-cutset for Constraint

optimization (Larrosa and Dechter, 2002)


The idea of super-buckets

Larger super-buckets (cliques) =>more time but less space

Complexity:1. Time: exponential in clique (super-bucket) size2. Space: exponential in separator size


Example


Super-Bucket Elimination, SBE(k)

Eliminate sets of variables such that:• individual eliminations are too costly in

space (namely, each variable in the set has degree larger than k)

• the join degree is lower than k


Sep-based time-space tradeoff Let T be a tree-decomposition of hypergraph H. Let

s_0,s_1,...,s_n be the sizes of the separators in T, listed in strictly descending order. With each separator size s_i we associate a secondary tree decomposition T_i, generated by combining adjacent nodes whose separator sizes are strictly greater than s_i.

We denote by r_i the largest set of variables in any cluster of T_i.

Note that as s_i decreases, r_i increase.

Theorem: The complexity of CTE when applied to each T_i is O( n exp(r_i)) time, and O( n exp(s_i)) space.


Complexity

Theorem: If R = (X,D,C) is a constraint network whose constraint graph has non-separable components of at most size r, then the super-bucket elimination algorithm, whose buckets are the non-separable components, is time exponential O(n exp(r)) and is linear in space.


Hybrids of hybrids hybrid(b_1,b_2): First, a tree-decomposition having separators bounded by b_1

is created, followed by application of the CTE algorithm, but each clique is processed by elim-cond(b_2). If c^*_{b_2} is the size of the maximum b_2-cutset in each clique of the b_1-tree-decomposition, the algorithm is space exponential in b_1 but time exponential in c^*_{b_2}.

Special cases: • hybrid(b_1,1): Applies cycle-cutset in each clique.• b_1 = b_2. For b=1, hybrid(1,1) is the non-separable components

utilizing the cycle-cutset in each component. The space complexity of this algorithm is linear but its time

complexity can be much better than the cycle-cutsets cheme or the non-separable component approach alone.


Case study: combinatorial circuits: benchmark used for fault diagnosis and testing community

Problem: Given a circuit and its unexpected output, identify faulty components.

The problem can be modeled as a constraint optimization problem and

solved by bucket elimination.


Case study: C432

A circuit’s primal graphFor every gate we connect inputs and outputs

Join-tree of c432Seperator size is 23


Join-tree of C3540 (1719 vars)max sep size 89


Secondary trees for C432


Time-space tradeoffsTime/Space tradeoff Time is measured

by the maximum of the separator size and the cutset size and

space by the maximum separator size.


Constraint Optimization:Combinatorial Auction: Bucket-elimination vs Search

b1 b2

b3

b4b5

b6

Bucket-elimination = Dynamic programming

)6(|| :

)6,5(),6,5(||, :

)6,4(||,, :

)3,6(||,, :

,, :

,, :

566

4256,55

346,45,44

134,36,33

13,26,11

25,26,22

hwb

hhwCb

hwCCb

hwCCb

wCCb

wCCb

Bucket-elimination: In a bucket sum costs and maximize over constrained assignments:

b2

b1

b3

b4

b5

b6

Search: Branch and Bound or Best-first search.


Recursive-search: a linear space search guided by a tree-decomposition

Given a tree network, we identify a node x_1 which, when removed, generates two subtrees of size n/2 (approximately).

T_n is the time to solve a binary tree starting at x_1. T_n obeys recurrence• T_n = k 2 T_n/2, T_1 = k

We get:• T_n = n k^{logn +1}

Given a tree-decomposition having induced-width w* this generalize to recursive conditioning of tree-decompositions:• T_n = n k^({w*+1} log n)

because the number of values k is replaced by th enumber of tuples k^w*


Road Map


Cutset decomposition AND/OR cutset decomposition Super-bucket and super-clusters Time-space curves


OR Conditioning

A=yellow A=green

B=red B=blue B=red B=blueB=green B=yellow

C

K

G

L

D

FH

M

J

E

C

K

G

L

D

FH

M

J

E

C

K

G

L

D

FH

M

J

E

C

K

G

L

D

FH

M

J

E

C

K

G

L

D

FH

M

J

E

C

K

G

L

D

FH

M

J

E


• Condition on some of the variablesA

C

B K

G

L

D

FH

M

J

E



AND/OR w-cutset

A

C

B K

G L

D F

H

M

J

E

AC

B K

G

L

D

FH

M

J

E

A

C

B K

G L

D F

H

M

J

E

C

B K

G

L

D

FH

M

J

E

3-cutset

A

C

B K

G L

D F

H

M

J

E

C

K

G

L

D

FH

M

J

E

2-cutset

A

C

B K

G L

D F

H

M

J

E

L

D

FH

M

J

E

1-cutset


Selecting an AND/OR w-cutset

AC

B K

G

L

D

FH

M

J

E

AC

B K

G

L

D

FH

M

J

E

AC

B K

G

L

D

FH

M

J

E

pseudo tree 1-cutset treegrahpical model


Searching AND/OR Graphs

AO(j): searches depth-first, cache i-context • j = the max size of a cache table (i.e. number

of variables in a context)

j=0 j=w*

Space: O(n)

Time: O(exp(w* log n))

Space: O(exp w*)

Time: O(exp w*)Space: O(exp(j) )

Time: O(exp(m_j+j)

m_j - AO depth of j-cutset

j


Conceptual difference

Traditional (w) cutset:• Minimize the number of nodes

• Improve by organizing in AND/OR tree

AND/OR (w) cutset:• Minimize depth of AND/OR cutset tree


AND/OR (w) cutset properties THEOREM: Given a graphical model,

|Cutset | ≥ depth(Cutset) ≥ depth(AO-Cutset)

The inequalities are often strict.

Cutset = minimal cardinality (w) cutsetdepth(Cutset) = minimal depth over pseudo trees for

Cutset

AO-Cutset = minimal depth AND/OR (w) cutset


Quality of start pseudo trees

0

10

20

30

40

50

60

70

80

90

1 2 3 4 5 6 7 8 9 10 11

i - bounds the complexity of the conditioned subproblem

f(i)

- bo

un

ds

the

com

ple

xity

of

th

e en

tire

pro

ble

m

f(i) = i + |Cutset|

f(i) = i + depth(Cutset)

f(i) = i + depth(AO-Cutset)

CPCS 422 network


Improved caching scheme

context(Xk) = [X1X2…Xk]

i-bound < k

X1

A

XK-i

Xk

XK-i+1

context(Xk) = [Xk-i+1…Xk] in conditioned subproblem

Cache for Xk is purged for every new instantiation of Xk-i

X1

A

XK-i

Xk

XK-i+1


Experimental results Measures

• Time

Networks• Random, Genetic Linkage

Algorithms• AO(i) – AO with caching full contexts• (No Caching on Cutset) + (AO Search)• (Caching on Cutset) + (AO Search)• (Caching on Cutset) + (BE)


Random networks (Time)

N=40, K=3, P=2, 20 instances, w*=7

i-bound Algorithms depth Time(sec)

MinFill MinDepth MinFill MinDepth

1 AO(i) 12 9 610.14 27.12

(No Caching on Cutset) + (AO Search) 174.53 8.75

(Caching on Cutset) + (AO Search) 67.99 7.61

(Caching on Cutset) + (BE) 16.95 2.18

3 AO(i) 7 6 71.68 8.13




5 AO(i) 4 3 11.28 2.77





Random networks (Time)

N=60, K=3, P=2, 20 instances, w*=11



6 (No Caching on Cutset) + (AO Search) 7 6 159.79 63.01










Genetic Linkage Networks

EA4 - N=1173, K=5, w*=15



6 AO(i) 23 21 10.0 103.4




9 AO(i) 18 17 3.3 9.3




13 AO(i) 3 8 2.0 5.9





Time-Space complexity of of w-cutset

Space: O(exp(w))• W-cutset: a set that when removed the induced-

width is w.

• c(w): size of w-cutset.

• m(w): depth of AO w-cutset

Time: O(exp(w+c(w))) on OR space Time: O(exp(w+m(w))) on AND/OR space

iimtw

iictw

twwcwicicc

)(*

,)(*

**)(*...)(...)2(21)1(

Documents

Hybrid of search and inference: time-space tradeoffs chapter 10