Organizing Open Online Computational Problem Solving Competitions

By: Ahmed Abdelmeged

•In 2011, researchers from the Harvard Catalyst Project were investigating the potential of crowdsourcing genome-sequencing algorithms.

•So, they collected few million sequencing problems and developed an electronic judge that evaluates sequencing algorithms by how well they solve these problems.

•And, they set up a two-week open online competition on TopCoder with a total prize pocket of $6000.

•The results were astounding!

•“... A two-week online contest ... produced over 600 submissions ... . Thirty submissions exceeded the benchmark performance of the US National Institutes of Health’s MegaBLAST. The best achieved both greater accuracy and speed (1,000 times greater).”

-- Nature Biotechnology, 31(2):pp. 108–111, 2013.

•We want to lower the barrier to entry for establishing such competitions by having “meaningful” competitions where participants assist the admin in evaluating their peers.

Thesis Statement

•“Semantic games of interpreted logic sentences provide a useful foundation to organize computational problem solving communities.”

Semantic Games

•A Semantic Game (SG) is a constructive debate of the correctness of an interpreted logic sentence (a.k.a claim) between two distinguished parties: the verifier which asserts that the claim holds, and the falsifier which asserts that the claim does not hold.

A Two-Party, SG-Based MAX-SAT Competition (I)

•Participants develop functions to:

•Provide side preference.

•Provide values for quantified variables based on values of variables in scope.

∀φ ∈ CNFs ∃v ∈ assignments(φ)∀f ∈ assignments(φ). fsat(f,φ)≤fsat(v,φ)

A Two-Party, SG-Based MAX-SAT Competition (II)

•Admin chooses sides for players based on their side preference.

•Let Pv be the verifier and Pf be the falsifier.

∀φ ∈ CNFs ∃v ∈ assignments(φ)∀f ∈ assignments(φ). fsat(f,φ)≤fsat(v,φ)

A Two-Party, SG-Based MAX-SAT Competition

(III)

•Admin gets value provided by Pf for φ.

•Admin checks φ ∈ CNFs. If false, Pf loses.

•Admin gets value provided by Pv for v.

•Admin checks v ∈ assignments(φ). If false, Pv loses.

∀φ ∈ CNFs ∃v ∈ assignments(φ)∀f ∈ assignments(φ). fsat(f,φ)≤fsat(v,φ)

A Two-Party, SG-Based MAX-SAT Competition

(IV)

•Admin gets value provided by Pf for f.

•Admin checks f ∈ assignments(φ). If false, Pf loses.

•Admin evaluates fsat(f,φ)≤fsat(v,φ). If true Pv wins, otherwise Pf wins.

∀φ ∈ CNFs ∃v ∈ assignments(φ)∀f ∈ assignments(φ). fsat(f,φ)≤fsat(v,φ)

Rationale (I)

•Controllable admin overhead.

∀φ ∈ CNFs ∃v ∈ assignments(φ). satisfies-max(v,φ)

∀φ ∈ CNFs ∃v ∈ assignments(φ)∀f ∈ assignments(φ). fsat(f,φ)≤fsat(v,φ)

Rationale (II)

•Correct: there is a winning strategy for verifiers of true claims and falsifiers of false claims. Regardless of the opponent’s actions.

Rationale (III)

•Objective.

•Systematic.

•Learning chances.

Challenges (I)

•Participants must take opposing sides!

•Neutrality is lost with forcing.

Challenges (II)•Scaling to N-Party Competition using a tournament, yet:

• Avoid Collusion Potential especially in the context of open online competitions where Sybil identities are common and games are too fast to spectate!

• Ensure that participants get the same chance.

Semantic Game Tournaments

Tournament Design

•Scheduler:

•Neutral.

•Ranking function:

•Correct and anonymous.

•Can mask scheduler deficiencies.

Ranking Functions

•Input : beating function representing output of several games.

•Output: a total preorder of participants.

Beating Functions (of SG

Tournaments)•bP(pw, pl, swc, slc, sw) : sum of all

gains of pw against pl while pw

choosing side swc , pl choosing side slc and pw taking side sw.

•More complex.

Ranking Functions (Correctness)

•Non-Negative Regard for Wins.

•Non-Positive Regard for Losses.

Non-Negative Regard For Wins

(NNRW)

PxWins

Faults

Additional winscannot worsen Px’s

rank w.r.t. other participants.

Non-Positive Regard For Losses

(NPRL)

PxWins

Faults

Additional faultscannot improve Px’s

rank w.r.t. other participants.

Implies:

Ranking Functions (Anonymity)

•Output ranking is independent of participant identities.

•Ranking function ignores participants’ identities.

•Participants also ignore their opponents’ identities.

Limited Collusion Effect

•Slightly weaker notion than anonymity.

•What you want in practice.

•A participant Py can choose to lose on purpose against another participant Px, but that won’t make Px get ahead of any other participant Pz.

Limited Collusion Effect (LCE)

PxWins

Faults

Games outside Px’s control cannot worsen Px’s rank w.r.t. other

participants.

Discovery

•A useful design principle for ranking functions.

•Under NNRW, NPRL : LCE = LFB

•LFB is quite unusual to have.

•LFB lends itself to implementation.

Locally Fault Based (LFB)

PxWins

Faults

Relative rank of Px and Py depends

only on faults made by either Px or Py.

PyFaults

Wins

Fault Counting Ranking Function

•Players are ranked according to the number of faults they make. The less the number of faults the higher the rank.

•Satisfies the NNRW, NPRL, LFB and LCE properties.

Semantic Game Tournament Design

•For every pair of players:

•If choosing different sides, play a single SG.

•If choosing same sides, play two SGs where they switch sides.

Neutrality

•Each player plays nv+ n

f - 1 SGs in

their chosen side, those are the only games it may make faults.

Related Work

•Rating and Ranking Functions

•Tournament Scheduling

•Match-Level Neutrality

Rating and Ranking Functions (I)

•Dominated by heuristic approaches

•Elo ratings.

•Who’s #1?

•There are axiomatization of rating functions in the field of Paired Comparison Analysis.

•LCE not on radar.

•Independence of Irrelevant Matches (IIM) is frowned upon.

Rating and Ranking Functions (II)

• Rubinstein[1980]:

• points system (winner gets a point) characterized as:

• Anonymity : ranks are independent of the names of participants.

• Positive responsiveness to the winning relation which means that changing the results of a participant p from a loss to a win, guarantees that p’s rank would improve.

• IIM: relative ranking of two participants is independent of matches in which neither is involved.

• “beating functions” are restricted to complete, asymmetric relations.

Tournament Scheduling

•Neutrality is off radar.

•Maximizing winning chances for certain players.

•Delayed confrontation.

Match-Level Neutrality

•Dominated by heuristic approaches

•Compensation points.

•Pie rule.

Conclusion

•“Semantic games of interpreted logic sentences provide a useful foundation to organize computational problem solving communities.”

Future Work

•Problem decomposition labs.

•Social Computing.

•Evaluating Thoroughness.

Questions?

Thank You!