Designing Example Critiquing Interaction

Designing Example Critiquing Interaction

Boi FaltingsPearl PuMarc TorrensPaolo Viappiani

IUI 2004, Madeira, Portugal – Wed Jan 14, 2004

LIA HCI

Wed Jan 14, 2004 Designing Example Critiquing Interaction 2

Outline

Introduction Stimulating expression of preferences Guaranteeing optimal solutions Conclusion


Motivation

Many real word applications require people to select a most preferred outcome from a large set of possibilities (electronic catalogs)

Users are usually unable to correctly state their preferences up front

People are greatly helped by seeing examples of actual solutions example critiquing


Mixed Initiative Interaction

initial preference

the system shows K solutions

The user critiques the solutions stating a new preference

The user picks the final choice


An implementation: reality

user critiques existing solutions

trade-off between different criteria


What to show?

Standard approach show the best solutions assumption: user model is complete and

accurate Does not in general stimulate new

preferences


New approach

Display_set = stimulate_set + optimal_set


What to show?

Stimulate set = solutions that make the user aware of attributes

diversity have high probability to become optimal if

new preferences are stated Optimal set = solutions that

are optimal given the current preferences


Outline

Introduction Stimulating expression of

preferences Guaranteeing optimal solutions Conclusion


Stimulating new preferences

Pareto optimality general concept does not involve weights

Dominated solution can become Pareto optimal if new preferences are stated show solutions that have higher

probability of becoming Pareto optimal


Dominance relation and Pareto optimality

P1 P2S1 0.3 0.4S2 0.4 0.2S3 0.6 0.5S4 0.3 0.9S5 0.9 0.7

3

6

3

9

6 9

Penalty table, 2 preferences

s1

s2

s3

s4

s5

S1 and S2 are Pareto optimal

S3 is dominated by S1 and S2

S4 is dominated by S1

S5 is dominated by S1, S2, S3.P1

P2


Pareto Optimal Filters

Estimate the probability that a dominated solution can become Pareto optimal when new preferences are stated

Different Pareto-filters: counting filter attribute filter probabilistic filter


Counting filter

We count the number of dominators

S1 and S2 are currently “optimal”

S4 more promising than S3 and S5

P1 P2 FilterS1 0.3 0.4 0S2 0.4 0.2 0S3 0.6 0.5 2S4 0.3 0.9 1S5 0.9 0.7 3

Counting Filter: number of Dominators


A new preference is added

New column with penalties S4 becomes Pareto optimal even if the new penalty

(0.6) is worse than for S3 (0.5) and S5 (0.4)

P1 P2 P3 Pareto Optimal

S1 0.3 0.4 0.7 YesS2 0.4 0.2 0.3 YesS3 0.6 0.5 0.5 NoS4 0.3 0.9 0.6 YesS5 0.9 0.7 0.4 No

The counting filter predict that S4 has better chances to become P.O. when a new preference is added.


Hasse diagrams

1 2

34

5

User Model={p1,p2}

Pareto Optimal

User Model={p1,p2, p3}

1 2

3

4

5

Adding preferences: Pareto optimal set grows, dominance relation becomes sparse


Attribute filter

Solution: n attributes: A1,..,An

D1,..,Dn domains for A1,..,An

a solution is a complete assignment Preferences modeled as penalty functions

defined on attribute domains Look at the attribute space

if two values are the same, any penalty function defined on these values will be the same


Price M2 Location TransportS1 650 28 West BusS2 700 24 North TramwayS3 700 24 West Bus

Attribute filter: motivation

S2 and S3 are both dominated by S1

If we add new preference on Location if North is preferred S2 will be Pareto

Optimal on Transport if Tramway is preferred to Bus then S2

will be P.O. S3 will always be dominated!!

Preferences: on price (to minimize), on M2 (to maximize)


Attribute Filter

For the new preference, dominated solution s must have lower penalty than all dominant solutions for discrete domain, attribute values must

be different for continuous domains, consider extreme

values


Probabilistic filter Directly estimate probability of becoming P.O. The bigger the difference on a specific attribute, the

more likely the penalties will be different

1 1

domain

pe

na

lty

pe

na

lty

domain


Experiments

Database of actual accommodation offers (room for rent, studios, apartments)

Random datasets 11 attributes of which

4 continuous (price, duration, square meters, distance to university)

7 discrete (kitchen, kitchen type, bathroom, public transportation, ..)


Results (accommodation dataset)


Results (random dataset)

0

0.1

0.2

0.3

0.4

0.5

0.6

1 2 3 4 5 6

CountProbAttRandom

Ave

rage

fra

ctio

n o

f co

rrec

t p

red

icti

ons

number of preferences known


Outline



Modelling True preference model P* (unknown)

P*={p*1, p*

2, .., p*

k}

st: target solution Estimated through a model P

P={p1,..,pk} pi are built-in standard penalty functions assume limited difference between p and p*

Penalty functions p

i(a

k): d

k -> R

write pi(s) instead of p

i(a

j(s))


Selecting displayed solutions

Dominance filters Utilitarian filters Egalitarian filters


Optimal Set Filters Properties

We want..

1. To show a limited number of solutions each filter selects k solutions to display

2. To ensure that a Pareto-optimal solution in D is Pareto-optimal in S

each filter satisfies this dominance filter (by definition), Utilitarian and

Egalitarian (theorem)


Optimal Set Filters Properties

3. To include target solutions only if target solution is included the user can

choose it! probability to include the target solution in D

depends on filter.

Assumption (1-ε)pi ≤ pi* ≤ (1+ε)pi


Dominance filter

Display k solutions that are not dominated by another one


Dominance filter: target solution

Plot of probability of target solution being included in D, as function of number of preferences |P|=3,..,12 K=30, 60 m=778, 6444


Utilitarian filter

We minimize the un-weighted sum of penalties

Efficiently computed by “branch & bound”

Pp

i

i

spsC )()(


Utilitarian filter: probability to find target solution

Does not depend on m, the number of total solutions (proved analytically)

Better than dominator filter


Egalitarian filter

Minimize F(s) In case of equality, use lexicographic order:(0.4, 0.2) preferred to (0.4, 0.4)

Target solution inclusion probability similar to that of the Utilitarian filter.

)(max)( spsF iPpi


Robustness against violated assumption

Fraction of PO solutions shown within the best k ones

Egalitarian, utilitarian filter


Outline



Conclusion

Optimal and stimulation set Example critiquing on firmer

mathematical ground Suggestions to system developer How to compensate an

incomplete/inaccurate user model Experimental evaluation on real and

random problems


Questions



Counting filter works already fairly well Attribute filter works very well when only

1 or 2 preferences are missing, but generally probabilistic is the best

Impact of correlation between attributes can affect performance

Pareto Filters: conclusions

Complexity

Counting Attribute ProbabilisticRandom


Attribute filter/2

Continuous domains: best values are the extremes

assumption: preference functions are monotonic

1OR

domain

pe

na

lty


1

θθ

1

domain domain

ai(o1) ai(o1)ai(o2) ai(o2)

1

θ

domain

ai(o1) ai(o2)

m

domain

ai(o1) ai(o2)


θθ

1

domain domainai(o)

gilisi

1

θdomain

ai(o) li

domain

ai(o) li

1

ai(o1)-t θ-t

m1

m2

ai(o)θ+t



Theorem

Given a set of m solutions S={s1,..,sm} and a set of penalties {p1,..,pd}

Let S’ be the best k solutions according to the utilitarian filter

A solution s in S’ not dominated by any other of S’, is Pareto Optimal in S.


Simplified Apartment Domain

A very simple example: A={Location, Rent, Rooms} DLocation={Centre, North, South, East, West}

DRent={x|x integer x>0}

DRooms={1,2,3..}

Preferences: location should be centre and rent less than 500


Penalty functions

P1:= if (Location==centre) then 0 else 1

P2:= If (Rent > 500) then K*(Rent-500) Else 0


Electronic catalogues

K attributes: A1,..,Ak

D1,..,Dk domains for A1,..,Ak

a solution is a complete assignment write aj(s), value of S for attribute j

Solution set S is a subset of D1 x D2 x D3 x D4 x ...

Preferences modeled as penalty functions defined on attribute domains


Counting filter*The Dominator set for a solution s

1, is the subset of S of

solution that dominates s1.

The counting filter orders

solutions on the size of

the dominator set.S

d(s

1)

s1


Probabilistic filter Directly estimate probability of becoming P.O. The bigger the difference on a specific attribute, the

more likely the penalties will be different

1

domain

pe

na

lty

Documents

Designing Example Critiquing Interaction