An Interactive System for Hiring & Managing Graduate Teaching Assistants Ryan Lim Venkata Praveen Guddeti Berthe Y. Choueiry Constraint Systems Laboratory

An Interactive System for Hiring & Managing Graduate Teaching Assistants

Ryan Lim

Venkata Praveen Guddeti

Berthe Y. Choueiry

Constraint Systems Laboratory

University of Nebraska-Lincoln

Outline Task & Motivation System Architecture & Interfaces Scientific aspects

Problem Modeling Problem Solving Comparing & Characterizing Solvers

Motivation revisited & Conclusions

Task Hiring & managing GTAs as instructors + graders

Given • A set of courses• A set of graduate teaching assistants• A set of constraints that specify allowable assignments

Find a consistent & satisfactory assignment• Consistent: assignment breaks no (hard) constraints• Satisfactory: assignment maximizes

1. number of courses covered 2. happiness of the GTAs

Often, number of hired GTAs is insufficient

Motivation Context

“Most difficult duty of a department chair” [Reichenbach, 2000]

Assignments done manually, countless reviews, persistent inconsistencies

Unhappy instructors, unhappy GTAs, unhappy students

Observation Computers are good at maintaining consistency Humans are good at balancing tradeoffs

Our solution An online, constraint-based system With interactive & automated search mechanisms




System Architecture

1. Web-interface for applicants

Password ProtectedAccess for GTAs

http://cse.unl.edu/~gta

Cooperative, hybrid Search Strategies

Other structured,semi-structured,

orunstructured DBs

In progress

Visualization widgets

Password ProtectedAccess for Managerhttp://cse.unl.edu/~gta

2. Web-interface for manager View / edit GTA records Setup classes Specify constraints Enforce pre-assignments

Local DB

3. A local relational database Graphical selective queries

Interactive Search

Automated SearchHeuristic BT

Stochastic LSMulti-agent Search

Randomized BT 4. Drivers for Interactive assignments Automated search algorithms

GTA interface: Preference Specification

Manager interface: TA Hiring & Load




Constraint-based Model Variables

Grading, conducting lectures, labs & recitations Values

Hired GTAs (+ preference for each value in domain) Constraints

Unary: ITA certification, enrollment, time conflict, non-zero preferences, etc.

Binary (Mutex): overlapping courses Non-binary: same-TA, capacity, confinement

Objective longest partial and consistent solution (primary criterion) while maximizing GTAs’ preferences (secondary criterion)




Problem Solving Interactive decision making

Seamlessly switching between perspectives Propagates decisions (MAC)

Automated search algorithms Heuristic backtrack search (BT) Stochastic local search (LS) Multi-agent search (ERA) Randomized backtrack search (RDGR) Future: Auction-based, GA, MIP, LD-search, etc.

On-going: Cooperative/hybrid strategies

Manager interface: Interactive Selection

Dual perspectiveTask-centered view Resource-centered view

Heuristic BT Search Since we don’t know, a priori, whether instance is

solvable, tight, or over-constrained Modified basic backtrack mechanism to deal with this situation

We designed & tested various ordering heuristics: Dynamic LD was consistently best

Branching factor relatively huge (30) Causes thrashing, backtrack never reaches early variables

24 hr: 51 (26%)

1 min: 55 (20%)

Max depth: 57

Shallowest level reached by BT after …

Nu

mb

er

of

varia

ble

s: 6

9

Depth of the tree: 69

Stochastic Local Search Hill-climbing with min-conflict heuristic Constraint propagation:

To handle non-binary constraints (e.g., high-arity capacity constraints)

Greedy: Consistent assignments are not undone

Random walk to avoid local maxima Random restarts to recover from local

maxima

Multi-Agent Search (ERA) [Liu et al. 02]

“Extremely” decentralized local search Agents (variables) seek to occupy best positions (values) Environment records constraint violation in each position of an

agent given positions of other agents Agents move, egoistically, between positions according to

reactive Rules

Decisions are local An agent can always kick other agents from a favorite position

even when value of ‘global objective function’ is not improved ERA appears immune to local optima

Lack of centralized control Agents continue to kick each other Deadlock appears in over-constrained problems

Randomized BT Search Random variable/value selection allows

BT to visit a wider area of the search space [Gomes et al. 98]

Restarts to overcome thrashing

Walsh proposed RGR [Walsh 99]

Our strategy, RDGR, improves RGR with dynamic choice of cutoff values for the restart strategy [Guddeti & Choueiry 04]

Optimizing solutions

Primary criterion: solution length BT, LS, ERA, RGR, RDGR

Secondary criterion: preference values BT, LS, RGR, RDGR Criterion:

• Average preference• Geometric mean• Maximum minimal preference

More Solvers… Interactive decision making

Automated search algorithms BT, LS, ERA, RGR, RDGR. Future: Auction-based, GA, MIP, LD-

search, etc.

On-going: Cooperative / hybrid strategies




Comparing Solvers Using the same CSP encoding, students

implements solvers separately and competed for best results

Experience lead to the identification of behavioral criteria and regimes that characterize the performance of the various solvers in the context of GTAP

Characterizing Solvers General criteria

Stability, solution length, vulnerability to local optima, deadlock, thrashing, etc.

Tight but solvable instances ERA RDGR RGR BT LS

Over-constrained instances RDGR RGR BT ERA LS


Problem Modeling Problem Solving Comparing & Characterization Solvers


Motivation (revisited) “Most difficult duty of a department chair”

Keeps the manager in the decision loop while removing the need for tedious and error-prone manual assignments

Helps producing quick (3 weeks down to 2 days) and satisfactory (stable) assignments

Initially, assignments were manually done on paper Now, on-line data acquisition process Enabled department to streamline & standardize GTA selection,

hiring, and assignment

Overworked staff, unhappy GTAs Overjoyed staff (relieved from handling application forms and

massive paperwork) Enthusiastic anonymous online reviews from applicants

History & Evaluation System entirely built by students Modeling started in January 2001 Prototype system used since August 2001

Features improved and added as needs arised

No formal longitudinal study Since August 2003: 109 GTA users, 23

feedback responses Since April 2004, CSE implemented on-line

GTA evaluation by faculty on top of GTAAP

GTA Survey Results

0 3 6 9 12 15 18

Ease

Flexibility

Robustness

Ease

Flexibility

Robustness

Su

rvey q

uest

ion

s

Number of ratings

Bad OK Good Very good Excellent

GTA Online Feedback

23 responses

Navig

ati

on

Data

entr

y

Conclusions Integrated interactive & automated problem-

solving strategies Reduced the burden of the manager Lead to quick development of ‘stable’ solutions

Our efforts Helped the department Trained students in CP techniques Paved new avenues for research

• Cooperative, hybrid search• Visualization of solution space

<<< end of presentation

I welcome your questions

Please contact me for a live demo

# total agents : 65# agents involved in deadlock: 24# unused GTAs: 8

(colorless) agent in zero position

(colorful) agent in deadlock

Manager interface: Course Load Specification

Manager interface: Preassignment

Manager interface: Constraint Specification

Documents

An Interactive System for Hiring & Managing Graduate Teaching Assistants Ryan Lim Venkata Praveen Guddeti Berthe Y. Choueiry Constraint Systems Laboratory