Embed Size (px)
Citation preview
Entropy-Driven Online Active Learning for Interactive Calendar Management
Julie S WeberMartha E Pollack
2
Personal Assistants
bull Electronic meeting requests via email
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
562 messages this week
3
Personal Assistants
bull Key Requirement Knowledge of Scheduling Preferences
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Challenge Preference Elicitationin an Interactive Environment
4
Challenges of an Interactive Environment
bull Must be opportunistic
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Must balance efficient learning and user satisfaction
5
Example
Meeting RequestMeet Monday afternoon or Friday lunch
Monday 1pm
Monday 130
Monday 2pm
Monday 230
Monday 3pm
Friday 12pm
Friday 1230
Friday 1pm
Meeting Request
Meet Monday afternoon or Friday lunch
SolutionSet
Monday 130
Monday 230
Friday 1230
Presentation Set
6
Outline
bull EDALS Entropy-Driven Active Learning for
Schedulingbull Experimental Analysis
bull Conclusions amp Future Work
bull Calendar Management Systemsbull PTIME
bull Active Learning
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
7
Calendar Management Systems
bull Kozierok amp Maesbull Reinforcement Learning
bull Mitchell et al ndash CAP bull Berry et al ndash PTIME
bull General calendar management
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
8Personal Time Manager ndash PTIME
bull Part of CALO General Personal Assistant
bull Interactive Calendar Managementbull Key Learning Component PLIANT
bull Preference Learnerbull Active Learner
Berry et al 2005
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
10
Schedule FeaturesLocal Features Feature ValuesDay of week Mon tue wed thu fri
Start time Earlylate am lunch earlylate pm
Duration Short med-short med-long long
Type Colleague dean student talk other
Global Features (free time)Short free blocks None few some many
Medium free blocks None few some many
Long free blocks None few some many
Global Features (overlaps)ColleagueCollDeanStudTalkOther
None few some many
DeanDeanStudentTalkOther None few some many
StudentStudentTalkOther None few some many
TalkTalkOther None few some many
OtherOther None few some many
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
12
Active Learning
bull Yu (2005) ndash Selective Sampling for Rankingbull Selection driven by ambiguity metric
bull Gervasio et al (2005) ndash AL in PTIMEbull Comparison of static active learning
techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
13
Comparison of Active Learning Techniques
bull Directed techniquesbull Max Diversitybull Max Novelty
bull Undirected techniquesbull Greedybull ε ndash Greedybull Random
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Gervasio et al (2005)
(+ Best)(+ Best)
(+ Best)
14
New Selection Strategy
bull Undirected gt Directed (slightly)bull Evaluation criteria
bull Hypothesesbull Selection strategy influenced by
characteristics of solution setbull Combined technique may be effectivebull Diversity of solution set an interesting
metricbull Learning efficiency + user satisfaction
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
15EDALS Entropy-Driven Active Learning for Scheduling
bull Online algorithm that selects betweenbull High-diversity solution set =gt coarse-
grained learning =gt Max Diversity bull Low-diversity solution set =gt fine-
grained learning =gt ε ndash Greedy
Based on the entropy of the solution set
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
16
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Day Time Dur Type SFree
MFree
LFree
O12 O13
10000
00100
1000
10000
0010 0100 0010 1000
0100
00100
10000
1000
10000
0010 0100 0010 1000
0100
00100
00010
1000
10000
0010 0100 0010 1000
0100
00100
00001
0100
10000
0010 0100 0010 1000
0100
17
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Ff
Fff
i
f
Ef
E i
i
||
||1
Total average entropy
18
EDALS
Choose_Method(S)
1 E calculate_entropy(S)
2 If E le threshold
3 return Undirected(S)
4 Else
5 return Directed(S)
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
20
Experiments
1 Determine best threshold value for EDALS2 Determine best EDALS components 3 Compare EDALS against static selection strategies
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
2
Personal Assistants
bull Electronic meeting requests via email
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
562 messages this week
3
Personal Assistants
bull Key Requirement Knowledge of Scheduling Preferences
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Challenge Preference Elicitationin an Interactive Environment
4
Challenges of an Interactive Environment
bull Must be opportunistic
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Must balance efficient learning and user satisfaction
5
Example
Meeting RequestMeet Monday afternoon or Friday lunch
Monday 1pm
Monday 130
Monday 2pm
Monday 230
Monday 3pm
Friday 12pm
Friday 1230
Friday 1pm
Meeting Request
Meet Monday afternoon or Friday lunch
SolutionSet
Monday 130
Monday 230
Friday 1230
Presentation Set
6
Outline
bull EDALS Entropy-Driven Active Learning for
Schedulingbull Experimental Analysis
bull Conclusions amp Future Work
bull Calendar Management Systemsbull PTIME
bull Active Learning
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
7
Calendar Management Systems
bull Kozierok amp Maesbull Reinforcement Learning
bull Mitchell et al ndash CAP bull Berry et al ndash PTIME
bull General calendar management
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
8Personal Time Manager ndash PTIME
bull Part of CALO General Personal Assistant
bull Interactive Calendar Managementbull Key Learning Component PLIANT
bull Preference Learnerbull Active Learner
Berry et al 2005
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
10
Schedule FeaturesLocal Features Feature ValuesDay of week Mon tue wed thu fri
Start time Earlylate am lunch earlylate pm
Duration Short med-short med-long long
Type Colleague dean student talk other
Global Features (free time)Short free blocks None few some many
Medium free blocks None few some many
Long free blocks None few some many
Global Features (overlaps)ColleagueCollDeanStudTalkOther
None few some many
DeanDeanStudentTalkOther None few some many
StudentStudentTalkOther None few some many
TalkTalkOther None few some many
OtherOther None few some many
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
12
Active Learning
bull Yu (2005) ndash Selective Sampling for Rankingbull Selection driven by ambiguity metric
bull Gervasio et al (2005) ndash AL in PTIMEbull Comparison of static active learning
techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
13
Comparison of Active Learning Techniques
bull Directed techniquesbull Max Diversitybull Max Novelty
bull Undirected techniquesbull Greedybull ε ndash Greedybull Random
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Gervasio et al (2005)
(+ Best)(+ Best)
(+ Best)
14
New Selection Strategy
bull Undirected gt Directed (slightly)bull Evaluation criteria
bull Hypothesesbull Selection strategy influenced by
characteristics of solution setbull Combined technique may be effectivebull Diversity of solution set an interesting
metricbull Learning efficiency + user satisfaction
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
15EDALS Entropy-Driven Active Learning for Scheduling
bull Online algorithm that selects betweenbull High-diversity solution set =gt coarse-
grained learning =gt Max Diversity bull Low-diversity solution set =gt fine-
grained learning =gt ε ndash Greedy
Based on the entropy of the solution set
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
16
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Day Time Dur Type SFree
MFree
LFree
O12 O13
10000
00100
1000
10000
0010 0100 0010 1000
0100
00100
10000
1000
10000
0010 0100 0010 1000
0100
00100
00010
1000
10000
0010 0100 0010 1000
0100
00100
00001
0100
10000
0010 0100 0010 1000
0100
17
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Ff
Fff
i
f
Ef
E i
i
||
||1
Total average entropy
18
EDALS
Choose_Method(S)
1 E calculate_entropy(S)
2 If E le threshold
3 return Undirected(S)
4 Else
5 return Directed(S)
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
20
Experiments
1 Determine best threshold value for EDALS2 Determine best EDALS components 3 Compare EDALS against static selection strategies
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
3
Personal Assistants
bull Key Requirement Knowledge of Scheduling Preferences
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Challenge Preference Elicitationin an Interactive Environment
4
Challenges of an Interactive Environment
bull Must be opportunistic
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Must balance efficient learning and user satisfaction
5
Example
Meeting RequestMeet Monday afternoon or Friday lunch
Monday 1pm
Monday 130
Monday 2pm
Monday 230
Monday 3pm
Friday 12pm
Friday 1230
Friday 1pm
Meeting Request
Meet Monday afternoon or Friday lunch
SolutionSet
Monday 130
Monday 230
Friday 1230
Presentation Set
6
Outline
bull EDALS Entropy-Driven Active Learning for
Schedulingbull Experimental Analysis
bull Conclusions amp Future Work
bull Calendar Management Systemsbull PTIME
bull Active Learning
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
7
Calendar Management Systems
bull Kozierok amp Maesbull Reinforcement Learning
bull Mitchell et al ndash CAP bull Berry et al ndash PTIME
bull General calendar management
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
8Personal Time Manager ndash PTIME
bull Part of CALO General Personal Assistant
bull Interactive Calendar Managementbull Key Learning Component PLIANT
bull Preference Learnerbull Active Learner
Berry et al 2005
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
10
Schedule FeaturesLocal Features Feature ValuesDay of week Mon tue wed thu fri
Start time Earlylate am lunch earlylate pm
Duration Short med-short med-long long
Type Colleague dean student talk other
Global Features (free time)Short free blocks None few some many
Medium free blocks None few some many
Long free blocks None few some many
Global Features (overlaps)ColleagueCollDeanStudTalkOther
None few some many
DeanDeanStudentTalkOther None few some many
StudentStudentTalkOther None few some many
TalkTalkOther None few some many
OtherOther None few some many
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
12
Active Learning
bull Yu (2005) ndash Selective Sampling for Rankingbull Selection driven by ambiguity metric
bull Gervasio et al (2005) ndash AL in PTIMEbull Comparison of static active learning
techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
13
Comparison of Active Learning Techniques
bull Directed techniquesbull Max Diversitybull Max Novelty
bull Undirected techniquesbull Greedybull ε ndash Greedybull Random
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Gervasio et al (2005)
(+ Best)(+ Best)
(+ Best)
14
New Selection Strategy
bull Undirected gt Directed (slightly)bull Evaluation criteria
bull Hypothesesbull Selection strategy influenced by
characteristics of solution setbull Combined technique may be effectivebull Diversity of solution set an interesting
metricbull Learning efficiency + user satisfaction
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
15EDALS Entropy-Driven Active Learning for Scheduling
bull Online algorithm that selects betweenbull High-diversity solution set =gt coarse-
grained learning =gt Max Diversity bull Low-diversity solution set =gt fine-
grained learning =gt ε ndash Greedy
Based on the entropy of the solution set
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
16
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Day Time Dur Type SFree
MFree
LFree
O12 O13
10000
00100
1000
10000
0010 0100 0010 1000
0100
00100
10000
1000
10000
0010 0100 0010 1000
0100
00100
00010
1000
10000
0010 0100 0010 1000
0100
00100
00001
0100
10000
0010 0100 0010 1000
0100
17
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Ff
Fff
i
f
Ef
E i
i
||
||1
Total average entropy
18
EDALS
Choose_Method(S)
1 E calculate_entropy(S)
2 If E le threshold
3 return Undirected(S)
4 Else
5 return Directed(S)
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
20
Experiments
1 Determine best threshold value for EDALS2 Determine best EDALS components 3 Compare EDALS against static selection strategies
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
4
Challenges of an Interactive Environment
bull Must be opportunistic
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Must balance efficient learning and user satisfaction
5
Example
Meeting RequestMeet Monday afternoon or Friday lunch
Monday 1pm
Monday 130
Monday 2pm
Monday 230
Monday 3pm
Friday 12pm
Friday 1230
Friday 1pm
Meeting Request
Meet Monday afternoon or Friday lunch
SolutionSet
Monday 130
Monday 230
Friday 1230
Presentation Set
6
Outline
bull EDALS Entropy-Driven Active Learning for
Schedulingbull Experimental Analysis
bull Conclusions amp Future Work
bull Calendar Management Systemsbull PTIME
bull Active Learning
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
7
Calendar Management Systems
bull Kozierok amp Maesbull Reinforcement Learning
bull Mitchell et al ndash CAP bull Berry et al ndash PTIME
bull General calendar management
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
8Personal Time Manager ndash PTIME
bull Part of CALO General Personal Assistant
bull Interactive Calendar Managementbull Key Learning Component PLIANT
bull Preference Learnerbull Active Learner
Berry et al 2005
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
10
Schedule FeaturesLocal Features Feature ValuesDay of week Mon tue wed thu fri
Start time Earlylate am lunch earlylate pm
Duration Short med-short med-long long
Type Colleague dean student talk other
Global Features (free time)Short free blocks None few some many
Medium free blocks None few some many
Long free blocks None few some many
Global Features (overlaps)ColleagueCollDeanStudTalkOther
None few some many
DeanDeanStudentTalkOther None few some many
StudentStudentTalkOther None few some many
TalkTalkOther None few some many
OtherOther None few some many
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
12
Active Learning
bull Yu (2005) ndash Selective Sampling for Rankingbull Selection driven by ambiguity metric
bull Gervasio et al (2005) ndash AL in PTIMEbull Comparison of static active learning
techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
13
Comparison of Active Learning Techniques
bull Directed techniquesbull Max Diversitybull Max Novelty
bull Undirected techniquesbull Greedybull ε ndash Greedybull Random
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Gervasio et al (2005)
(+ Best)(+ Best)
(+ Best)
14
New Selection Strategy
bull Undirected gt Directed (slightly)bull Evaluation criteria
bull Hypothesesbull Selection strategy influenced by
characteristics of solution setbull Combined technique may be effectivebull Diversity of solution set an interesting
metricbull Learning efficiency + user satisfaction
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
15EDALS Entropy-Driven Active Learning for Scheduling
bull Online algorithm that selects betweenbull High-diversity solution set =gt coarse-
grained learning =gt Max Diversity bull Low-diversity solution set =gt fine-
grained learning =gt ε ndash Greedy
Based on the entropy of the solution set
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
16
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Day Time Dur Type SFree
MFree
LFree
O12 O13
10000
00100
1000
10000
0010 0100 0010 1000
0100
00100
10000
1000
10000
0010 0100 0010 1000
0100
00100
00010
1000
10000
0010 0100 0010 1000
0100
00100
00001
0100
10000
0010 0100 0010 1000
0100
17
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Ff
Fff
i
f
Ef
E i
i
||
||1
Total average entropy
18
EDALS
Choose_Method(S)
1 E calculate_entropy(S)
2 If E le threshold
3 return Undirected(S)
4 Else
5 return Directed(S)
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
20
Experiments
1 Determine best threshold value for EDALS2 Determine best EDALS components 3 Compare EDALS against static selection strategies
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
5
Example
Meeting RequestMeet Monday afternoon or Friday lunch
Monday 1pm
Monday 130
Monday 2pm
Monday 230
Monday 3pm
Friday 12pm
Friday 1230
Friday 1pm
Meeting Request
Meet Monday afternoon or Friday lunch
SolutionSet
Monday 130
Monday 230
Friday 1230
Presentation Set
6
Outline
bull EDALS Entropy-Driven Active Learning for
Schedulingbull Experimental Analysis
bull Conclusions amp Future Work
bull Calendar Management Systemsbull PTIME
bull Active Learning
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
7
Calendar Management Systems
bull Kozierok amp Maesbull Reinforcement Learning
bull Mitchell et al ndash CAP bull Berry et al ndash PTIME
bull General calendar management
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
8Personal Time Manager ndash PTIME
bull Part of CALO General Personal Assistant
bull Interactive Calendar Managementbull Key Learning Component PLIANT
bull Preference Learnerbull Active Learner
Berry et al 2005
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
10
Schedule FeaturesLocal Features Feature ValuesDay of week Mon tue wed thu fri
Start time Earlylate am lunch earlylate pm
Duration Short med-short med-long long
Type Colleague dean student talk other
Global Features (free time)Short free blocks None few some many
Medium free blocks None few some many
Long free blocks None few some many
Global Features (overlaps)ColleagueCollDeanStudTalkOther
None few some many
DeanDeanStudentTalkOther None few some many
StudentStudentTalkOther None few some many
TalkTalkOther None few some many
OtherOther None few some many
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
12
Active Learning
bull Yu (2005) ndash Selective Sampling for Rankingbull Selection driven by ambiguity metric
bull Gervasio et al (2005) ndash AL in PTIMEbull Comparison of static active learning
techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
13
Comparison of Active Learning Techniques
bull Directed techniquesbull Max Diversitybull Max Novelty
bull Undirected techniquesbull Greedybull ε ndash Greedybull Random
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Gervasio et al (2005)
(+ Best)(+ Best)
(+ Best)
14
New Selection Strategy
bull Undirected gt Directed (slightly)bull Evaluation criteria
bull Hypothesesbull Selection strategy influenced by
characteristics of solution setbull Combined technique may be effectivebull Diversity of solution set an interesting
metricbull Learning efficiency + user satisfaction
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
15EDALS Entropy-Driven Active Learning for Scheduling
bull Online algorithm that selects betweenbull High-diversity solution set =gt coarse-
grained learning =gt Max Diversity bull Low-diversity solution set =gt fine-
grained learning =gt ε ndash Greedy
Based on the entropy of the solution set
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
16
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Day Time Dur Type SFree
MFree
LFree
O12 O13
10000
00100
1000
10000
0010 0100 0010 1000
0100
00100
10000
1000
10000
0010 0100 0010 1000
0100
00100
00010
1000
10000
0010 0100 0010 1000
0100
00100
00001
0100
10000
0010 0100 0010 1000
0100
17
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Ff
Fff
i
f
Ef
E i
i
||
||1
Total average entropy
18
EDALS
Choose_Method(S)
1 E calculate_entropy(S)
2 If E le threshold
3 return Undirected(S)
4 Else
5 return Directed(S)
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
20
Experiments
1 Determine best threshold value for EDALS2 Determine best EDALS components 3 Compare EDALS against static selection strategies
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
6
Outline
bull EDALS Entropy-Driven Active Learning for
Schedulingbull Experimental Analysis
bull Conclusions amp Future Work
bull Calendar Management Systemsbull PTIME
bull Active Learning
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
7
Calendar Management Systems
bull Kozierok amp Maesbull Reinforcement Learning
bull Mitchell et al ndash CAP bull Berry et al ndash PTIME
bull General calendar management
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
8Personal Time Manager ndash PTIME
bull Part of CALO General Personal Assistant
bull Interactive Calendar Managementbull Key Learning Component PLIANT
bull Preference Learnerbull Active Learner
Berry et al 2005
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
10
Schedule FeaturesLocal Features Feature ValuesDay of week Mon tue wed thu fri
Start time Earlylate am lunch earlylate pm
Duration Short med-short med-long long
Type Colleague dean student talk other
Global Features (free time)Short free blocks None few some many
Medium free blocks None few some many
Long free blocks None few some many
Global Features (overlaps)ColleagueCollDeanStudTalkOther
None few some many
DeanDeanStudentTalkOther None few some many
StudentStudentTalkOther None few some many
TalkTalkOther None few some many
OtherOther None few some many
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
12
Active Learning
bull Yu (2005) ndash Selective Sampling for Rankingbull Selection driven by ambiguity metric
bull Gervasio et al (2005) ndash AL in PTIMEbull Comparison of static active learning
techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
13
Comparison of Active Learning Techniques
bull Directed techniquesbull Max Diversitybull Max Novelty
bull Undirected techniquesbull Greedybull ε ndash Greedybull Random
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Gervasio et al (2005)
(+ Best)(+ Best)
(+ Best)
14
New Selection Strategy
bull Undirected gt Directed (slightly)bull Evaluation criteria
bull Hypothesesbull Selection strategy influenced by
characteristics of solution setbull Combined technique may be effectivebull Diversity of solution set an interesting
metricbull Learning efficiency + user satisfaction
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
15EDALS Entropy-Driven Active Learning for Scheduling
bull Online algorithm that selects betweenbull High-diversity solution set =gt coarse-
grained learning =gt Max Diversity bull Low-diversity solution set =gt fine-
grained learning =gt ε ndash Greedy
Based on the entropy of the solution set
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
16
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Day Time Dur Type SFree
MFree
LFree
O12 O13
10000
00100
1000
10000
0010 0100 0010 1000
0100
00100
10000
1000
10000
0010 0100 0010 1000
0100
00100
00010
1000
10000
0010 0100 0010 1000
0100
00100
00001
0100
10000
0010 0100 0010 1000
0100
17
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Ff
Fff
i
f
Ef
E i
i
||
||1
Total average entropy
18
EDALS
Choose_Method(S)
1 E calculate_entropy(S)
2 If E le threshold
3 return Undirected(S)
4 Else
5 return Directed(S)
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
20
Experiments
1 Determine best threshold value for EDALS2 Determine best EDALS components 3 Compare EDALS against static selection strategies
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
7
Calendar Management Systems
bull Kozierok amp Maesbull Reinforcement Learning
bull Mitchell et al ndash CAP bull Berry et al ndash PTIME
bull General calendar management
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
8Personal Time Manager ndash PTIME
bull Part of CALO General Personal Assistant
bull Interactive Calendar Managementbull Key Learning Component PLIANT
bull Preference Learnerbull Active Learner
Berry et al 2005
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
10
Schedule FeaturesLocal Features Feature ValuesDay of week Mon tue wed thu fri
Start time Earlylate am lunch earlylate pm
Duration Short med-short med-long long
Type Colleague dean student talk other
Global Features (free time)Short free blocks None few some many
Medium free blocks None few some many
Long free blocks None few some many
Global Features (overlaps)ColleagueCollDeanStudTalkOther
None few some many
DeanDeanStudentTalkOther None few some many
StudentStudentTalkOther None few some many
TalkTalkOther None few some many
OtherOther None few some many
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
12
Active Learning
bull Yu (2005) ndash Selective Sampling for Rankingbull Selection driven by ambiguity metric
bull Gervasio et al (2005) ndash AL in PTIMEbull Comparison of static active learning
techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
13
Comparison of Active Learning Techniques
bull Directed techniquesbull Max Diversitybull Max Novelty
bull Undirected techniquesbull Greedybull ε ndash Greedybull Random
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Gervasio et al (2005)
(+ Best)(+ Best)
(+ Best)
14
New Selection Strategy
bull Undirected gt Directed (slightly)bull Evaluation criteria
bull Hypothesesbull Selection strategy influenced by
characteristics of solution setbull Combined technique may be effectivebull Diversity of solution set an interesting
metricbull Learning efficiency + user satisfaction
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
15EDALS Entropy-Driven Active Learning for Scheduling
bull Online algorithm that selects betweenbull High-diversity solution set =gt coarse-
grained learning =gt Max Diversity bull Low-diversity solution set =gt fine-
grained learning =gt ε ndash Greedy
Based on the entropy of the solution set
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
16
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Day Time Dur Type SFree
MFree
LFree
O12 O13
10000
00100
1000
10000
0010 0100 0010 1000
0100
00100
10000
1000
10000
0010 0100 0010 1000
0100
00100
00010
1000
10000
0010 0100 0010 1000
0100
00100
00001
0100
10000
0010 0100 0010 1000
0100
17
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Ff
Fff
i
f
Ef
E i
i
||
||1
Total average entropy
18
EDALS
Choose_Method(S)
1 E calculate_entropy(S)
2 If E le threshold
3 return Undirected(S)
4 Else
5 return Directed(S)
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
20
Experiments
1 Determine best threshold value for EDALS2 Determine best EDALS components 3 Compare EDALS against static selection strategies
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
8Personal Time Manager ndash PTIME
bull Part of CALO General Personal Assistant
bull Interactive Calendar Managementbull Key Learning Component PLIANT
bull Preference Learnerbull Active Learner
Berry et al 2005
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
10
Schedule FeaturesLocal Features Feature ValuesDay of week Mon tue wed thu fri
Start time Earlylate am lunch earlylate pm
Duration Short med-short med-long long
Type Colleague dean student talk other
Global Features (free time)Short free blocks None few some many
Medium free blocks None few some many
Long free blocks None few some many
Global Features (overlaps)ColleagueCollDeanStudTalkOther
None few some many
DeanDeanStudentTalkOther None few some many
StudentStudentTalkOther None few some many
TalkTalkOther None few some many
OtherOther None few some many
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
12
Active Learning
bull Yu (2005) ndash Selective Sampling for Rankingbull Selection driven by ambiguity metric
bull Gervasio et al (2005) ndash AL in PTIMEbull Comparison of static active learning
techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
13
Comparison of Active Learning Techniques
bull Directed techniquesbull Max Diversitybull Max Novelty
bull Undirected techniquesbull Greedybull ε ndash Greedybull Random
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Gervasio et al (2005)
(+ Best)(+ Best)
(+ Best)
14
New Selection Strategy
bull Undirected gt Directed (slightly)bull Evaluation criteria
bull Hypothesesbull Selection strategy influenced by
characteristics of solution setbull Combined technique may be effectivebull Diversity of solution set an interesting
metricbull Learning efficiency + user satisfaction
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
15EDALS Entropy-Driven Active Learning for Scheduling
bull Online algorithm that selects betweenbull High-diversity solution set =gt coarse-
grained learning =gt Max Diversity bull Low-diversity solution set =gt fine-
grained learning =gt ε ndash Greedy
Based on the entropy of the solution set
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
16
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Day Time Dur Type SFree
MFree
LFree
O12 O13
10000
00100
1000
10000
0010 0100 0010 1000
0100
00100
10000
1000
10000
0010 0100 0010 1000
0100
00100
00010
1000
10000
0010 0100 0010 1000
0100
00100
00001
0100
10000
0010 0100 0010 1000
0100
17
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Ff
Fff
i
f
Ef
E i
i
||
||1
Total average entropy
18
EDALS
Choose_Method(S)
1 E calculate_entropy(S)
2 If E le threshold
3 return Undirected(S)
4 Else
5 return Directed(S)
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
20
Experiments
1 Determine best threshold value for EDALS2 Determine best EDALS components 3 Compare EDALS against static selection strategies
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
10
Schedule FeaturesLocal Features Feature ValuesDay of week Mon tue wed thu fri
Start time Earlylate am lunch earlylate pm
Duration Short med-short med-long long
Type Colleague dean student talk other
Global Features (free time)Short free blocks None few some many
Medium free blocks None few some many
Long free blocks None few some many
Global Features (overlaps)ColleagueCollDeanStudTalkOther
None few some many
DeanDeanStudentTalkOther None few some many
StudentStudentTalkOther None few some many
TalkTalkOther None few some many
OtherOther None few some many
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
12
Active Learning
bull Yu (2005) ndash Selective Sampling for Rankingbull Selection driven by ambiguity metric
bull Gervasio et al (2005) ndash AL in PTIMEbull Comparison of static active learning
techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
13
Comparison of Active Learning Techniques
bull Directed techniquesbull Max Diversitybull Max Novelty
bull Undirected techniquesbull Greedybull ε ndash Greedybull Random
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Gervasio et al (2005)
(+ Best)(+ Best)
(+ Best)
14
New Selection Strategy
bull Undirected gt Directed (slightly)bull Evaluation criteria
bull Hypothesesbull Selection strategy influenced by
characteristics of solution setbull Combined technique may be effectivebull Diversity of solution set an interesting
metricbull Learning efficiency + user satisfaction
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
15EDALS Entropy-Driven Active Learning for Scheduling
bull Online algorithm that selects betweenbull High-diversity solution set =gt coarse-
grained learning =gt Max Diversity bull Low-diversity solution set =gt fine-
grained learning =gt ε ndash Greedy
Based on the entropy of the solution set
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
16
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Day Time Dur Type SFree
MFree
LFree
O12 O13
10000
00100
1000
10000
0010 0100 0010 1000
0100
00100
10000
1000
10000
0010 0100 0010 1000
0100
00100
00010
1000
10000
0010 0100 0010 1000
0100
00100
00001
0100
10000
0010 0100 0010 1000
0100
17
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Ff
Fff
i
f
Ef
E i
i
||
||1
Total average entropy
18
EDALS
Choose_Method(S)
1 E calculate_entropy(S)
2 If E le threshold
3 return Undirected(S)
4 Else
5 return Directed(S)
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
20
Experiments
1 Determine best threshold value for EDALS2 Determine best EDALS components 3 Compare EDALS against static selection strategies
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
10
Schedule FeaturesLocal Features Feature ValuesDay of week Mon tue wed thu fri
Start time Earlylate am lunch earlylate pm
Duration Short med-short med-long long
Type Colleague dean student talk other
Global Features (free time)Short free blocks None few some many
Medium free blocks None few some many
Long free blocks None few some many
Global Features (overlaps)ColleagueCollDeanStudTalkOther
None few some many
DeanDeanStudentTalkOther None few some many
StudentStudentTalkOther None few some many
TalkTalkOther None few some many
OtherOther None few some many
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
12
Active Learning
bull Yu (2005) ndash Selective Sampling for Rankingbull Selection driven by ambiguity metric
bull Gervasio et al (2005) ndash AL in PTIMEbull Comparison of static active learning
techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
13
Comparison of Active Learning Techniques
bull Directed techniquesbull Max Diversitybull Max Novelty
bull Undirected techniquesbull Greedybull ε ndash Greedybull Random
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Gervasio et al (2005)
(+ Best)(+ Best)
(+ Best)
14
New Selection Strategy
bull Undirected gt Directed (slightly)bull Evaluation criteria
bull Hypothesesbull Selection strategy influenced by
characteristics of solution setbull Combined technique may be effectivebull Diversity of solution set an interesting
metricbull Learning efficiency + user satisfaction
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
15EDALS Entropy-Driven Active Learning for Scheduling
bull Online algorithm that selects betweenbull High-diversity solution set =gt coarse-
grained learning =gt Max Diversity bull Low-diversity solution set =gt fine-
grained learning =gt ε ndash Greedy
Based on the entropy of the solution set
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
16
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Day Time Dur Type SFree
MFree
LFree
O12 O13
10000
00100
1000
10000
0010 0100 0010 1000
0100
00100
10000
1000
10000
0010 0100 0010 1000
0100
00100
00010
1000
10000
0010 0100 0010 1000
0100
00100
00001
0100
10000
0010 0100 0010 1000
0100
17
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Ff
Fff
i
f
Ef
E i
i
||
||1
Total average entropy
18
EDALS
Choose_Method(S)
1 E calculate_entropy(S)
2 If E le threshold
3 return Undirected(S)
4 Else
5 return Directed(S)
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
20
Experiments
1 Determine best threshold value for EDALS2 Determine best EDALS components 3 Compare EDALS against static selection strategies
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
CalendarManager
ConstraintReasoner
ActiveLearner
preferenceprofile
PreferenceLearner
PLIANT
ranked
presentation set
scheduling request
candidates
selectedcandidate
3
solutionset
7
4
5
2
1
6
Ranker
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
12
Active Learning
bull Yu (2005) ndash Selective Sampling for Rankingbull Selection driven by ambiguity metric
bull Gervasio et al (2005) ndash AL in PTIMEbull Comparison of static active learning
techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
13
Comparison of Active Learning Techniques
bull Directed techniquesbull Max Diversitybull Max Novelty
bull Undirected techniquesbull Greedybull ε ndash Greedybull Random
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Gervasio et al (2005)
(+ Best)(+ Best)
(+ Best)
14
New Selection Strategy
bull Undirected gt Directed (slightly)bull Evaluation criteria
bull Hypothesesbull Selection strategy influenced by
characteristics of solution setbull Combined technique may be effectivebull Diversity of solution set an interesting
metricbull Learning efficiency + user satisfaction
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
15EDALS Entropy-Driven Active Learning for Scheduling
bull Online algorithm that selects betweenbull High-diversity solution set =gt coarse-
grained learning =gt Max Diversity bull Low-diversity solution set =gt fine-
grained learning =gt ε ndash Greedy
Based on the entropy of the solution set
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
16
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Day Time Dur Type SFree
MFree
LFree
O12 O13
10000
00100
1000
10000
0010 0100 0010 1000
0100
00100
10000
1000
10000
0010 0100 0010 1000
0100
00100
00010
1000
10000
0010 0100 0010 1000
0100
00100
00001
0100
10000
0010 0100 0010 1000
0100
17
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Ff
Fff
i
f
Ef
E i
i
||
||1
Total average entropy
18
EDALS
Choose_Method(S)
1 E calculate_entropy(S)
2 If E le threshold
3 return Undirected(S)
4 Else
5 return Directed(S)
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
20
Experiments
1 Determine best threshold value for EDALS2 Determine best EDALS components 3 Compare EDALS against static selection strategies
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
12
Active Learning
bull Yu (2005) ndash Selective Sampling for Rankingbull Selection driven by ambiguity metric
bull Gervasio et al (2005) ndash AL in PTIMEbull Comparison of static active learning
techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
13
Comparison of Active Learning Techniques
bull Directed techniquesbull Max Diversitybull Max Novelty
bull Undirected techniquesbull Greedybull ε ndash Greedybull Random
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Gervasio et al (2005)
(+ Best)(+ Best)
(+ Best)
14
New Selection Strategy
bull Undirected gt Directed (slightly)bull Evaluation criteria
bull Hypothesesbull Selection strategy influenced by
characteristics of solution setbull Combined technique may be effectivebull Diversity of solution set an interesting
metricbull Learning efficiency + user satisfaction
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
15EDALS Entropy-Driven Active Learning for Scheduling
bull Online algorithm that selects betweenbull High-diversity solution set =gt coarse-
grained learning =gt Max Diversity bull Low-diversity solution set =gt fine-
grained learning =gt ε ndash Greedy
Based on the entropy of the solution set
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
16
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Day Time Dur Type SFree
MFree
LFree
O12 O13
10000
00100
1000
10000
0010 0100 0010 1000
0100
00100
10000
1000
10000
0010 0100 0010 1000
0100
00100
00010
1000
10000
0010 0100 0010 1000
0100
00100
00001
0100
10000
0010 0100 0010 1000
0100
17
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Ff
Fff
i
f
Ef
E i
i
||
||1
Total average entropy
18
EDALS
Choose_Method(S)
1 E calculate_entropy(S)
2 If E le threshold
3 return Undirected(S)
4 Else
5 return Directed(S)
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
20
Experiments
1 Determine best threshold value for EDALS2 Determine best EDALS components 3 Compare EDALS against static selection strategies
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
13
Comparison of Active Learning Techniques
bull Directed techniquesbull Max Diversitybull Max Novelty
bull Undirected techniquesbull Greedybull ε ndash Greedybull Random
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Gervasio et al (2005)
(+ Best)(+ Best)
(+ Best)
14
New Selection Strategy
bull Undirected gt Directed (slightly)bull Evaluation criteria
bull Hypothesesbull Selection strategy influenced by
characteristics of solution setbull Combined technique may be effectivebull Diversity of solution set an interesting
metricbull Learning efficiency + user satisfaction
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
15EDALS Entropy-Driven Active Learning for Scheduling
bull Online algorithm that selects betweenbull High-diversity solution set =gt coarse-
grained learning =gt Max Diversity bull Low-diversity solution set =gt fine-
grained learning =gt ε ndash Greedy
Based on the entropy of the solution set
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
16
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Day Time Dur Type SFree
MFree
LFree
O12 O13
10000
00100
1000
10000
0010 0100 0010 1000
0100
00100
10000
1000
10000
0010 0100 0010 1000
0100
00100
00010
1000
10000
0010 0100 0010 1000
0100
00100
00001
0100
10000
0010 0100 0010 1000
0100
17
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Ff
Fff
i
f
Ef
E i
i
||
||1
Total average entropy
18
EDALS
Choose_Method(S)
1 E calculate_entropy(S)
2 If E le threshold
3 return Undirected(S)
4 Else
5 return Directed(S)
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
20
Experiments
1 Determine best threshold value for EDALS2 Determine best EDALS components 3 Compare EDALS against static selection strategies
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
14
New Selection Strategy
bull Undirected gt Directed (slightly)bull Evaluation criteria
bull Hypothesesbull Selection strategy influenced by
characteristics of solution setbull Combined technique may be effectivebull Diversity of solution set an interesting
metricbull Learning efficiency + user satisfaction
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
15EDALS Entropy-Driven Active Learning for Scheduling
bull Online algorithm that selects betweenbull High-diversity solution set =gt coarse-
grained learning =gt Max Diversity bull Low-diversity solution set =gt fine-
grained learning =gt ε ndash Greedy
Based on the entropy of the solution set
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
16
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Day Time Dur Type SFree
MFree
LFree
O12 O13
10000
00100
1000
10000
0010 0100 0010 1000
0100
00100
10000
1000
10000
0010 0100 0010 1000
0100
00100
00010
1000
10000
0010 0100 0010 1000
0100
00100
00001
0100
10000
0010 0100 0010 1000
0100
17
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Ff
Fff
i
f
Ef
E i
i
||
||1
Total average entropy
18
EDALS
Choose_Method(S)
1 E calculate_entropy(S)
2 If E le threshold
3 return Undirected(S)
4 Else
5 return Directed(S)
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
20
Experiments
1 Determine best threshold value for EDALS2 Determine best EDALS components 3 Compare EDALS against static selection strategies
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
15EDALS Entropy-Driven Active Learning for Scheduling
bull Online algorithm that selects betweenbull High-diversity solution set =gt coarse-
grained learning =gt Max Diversity bull Low-diversity solution set =gt fine-
grained learning =gt ε ndash Greedy
Based on the entropy of the solution set
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
16
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Day Time Dur Type SFree
MFree
LFree
O12 O13
10000
00100
1000
10000
0010 0100 0010 1000
0100
00100
10000
1000
10000
0010 0100 0010 1000
0100
00100
00010
1000
10000
0010 0100 0010 1000
0100
00100
00001
0100
10000
0010 0100 0010 1000
0100
17
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Ff
Fff
i
f
Ef
E i
i
||
||1
Total average entropy
18
EDALS
Choose_Method(S)
1 E calculate_entropy(S)
2 If E le threshold
3 return Undirected(S)
4 Else
5 return Directed(S)
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
20
Experiments
1 Determine best threshold value for EDALS2 Determine best EDALS components 3 Compare EDALS against static selection strategies
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
16
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Day Time Dur Type SFree
MFree
LFree
O12 O13
10000
00100
1000
10000
0010 0100 0010 1000
0100
00100
10000
1000
10000
0010 0100 0010 1000
0100
00100
00010
1000
10000
0010 0100 0010 1000
0100
00100
00001
0100
10000
0010 0100 0010 1000
0100
17
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Ff
Fff
i
f
Ef
E i
i
||
||1
Total average entropy
18
EDALS
Choose_Method(S)
1 E calculate_entropy(S)
2 If E le threshold
3 return Undirected(S)
4 Else
5 return Directed(S)
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
20
Experiments
1 Determine best threshold value for EDALS2 Determine best EDALS components 3 Compare EDALS against static selection strategies
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
17
Solution Set Entropy
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
)1(log)1(||
1||
ij
f
jfijf fPfPE
i
ii
Entropy of a single feature
Ff
Fff
i
f
Ef
E i
i
||
||1
Total average entropy
18
EDALS
Choose_Method(S)
1 E calculate_entropy(S)
2 If E le threshold
3 return Undirected(S)
4 Else
5 return Directed(S)
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
20
Experiments
1 Determine best threshold value for EDALS2 Determine best EDALS components 3 Compare EDALS against static selection strategies
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
18
EDALS
Choose_Method(S)
1 E calculate_entropy(S)
2 If E le threshold
3 return Undirected(S)
4 Else
5 return Directed(S)
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
20
Experiments
1 Determine best threshold value for EDALS2 Determine best EDALS components 3 Compare EDALS against static selection strategies
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
20
Experiments
1 Determine best threshold value for EDALS2 Determine best EDALS components 3 Compare EDALS against static selection strategies
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
22
Performance Criteria
bull Spearmanrsquos Correlation Coefficientbull Difference between systemrsquos ranking
and userrsquos ranking of 100 schedules
bull User Satisfactionbull Rank of the best option in
presentation set compared to other feasible alternatives
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
23
Initial EDALS Experiment
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Loose3 - Spearman Correlation
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
24Static Selection Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
25EDALS Component Selection
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Spearman User Satisfaction
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
26EDALS vs Static Techniques
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
Seeded - SpearmanUnseeded - Spearman
Unseeded ndash User Satisfaction
Seeded ndash User Satisfaction
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
27
Conclusion
bull An online approach to active learning appears both learning-effective and user-satisfying in the scheduling preference-learning domain
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
28
Future Work
bull Human users
Calendar Assistants|PTIME|Active Learning|Algorithm|Results|Conclusions
bull Dynamic thresholds
bull Adjustable autonomybull Application to other domains
