Practical tips on making better predictions

Or, how to get to #1 on DAGGRE

“Who are you? And why are you presenting?”

• Scott Leibrand, age 32• B.S. in Cell & Molecular Biology• Advanced Services Architect for

Limelight Networks• Nope, no professional qualifications

(for forecasting world events, anyway)

• But, I worked my way up to #1 on the DAGGRE leaderboard, so here I am.

Overview• Be judicious• Be less wrong– than the next guy– than you were yesterday

• Gather and synthesize information– Act like a fox, not a hedgehog

• Share your reasoning• Prioritize• Trade well– Understand the scoring system, and trade accordingly

• Keep track of your positions (it’s easy)

Be Judicious

• The biggest mistake people make in forecasting is being overconfident.

• On DAGGRE, this manifests itself as large edits way outside the range of every else’s estimates, in the absence of new information.

• If you make a large edit like that, you’re throwing points away. If you’re confident the consensus is wrong, make a number of small edits. If someone partially reverses you, you’ll be able to accumulate a much larger position much more cheaply.

Be Less Wrong

• You’re going to be wrong. Just be less wrong.• Be less wrong than the next guy: If someone makes an

overconfident edit, (partially) reverse it.• Be less wrong than you were yesterday: don’t

unwaveringly stick to your previous position. If you come across new information, update your estimate.

• Be Bayesian: use new info to update your probabilities appropriately.

• There’s no penalty for changing your mind and reversing your own estimates.

Gather and synthesize information• You can’t make a good estimate without

considering all the relevant, available facts.• Read the comments and links others have

posted in the comment section.• Google makes searching easy. Google News

Alerts are also essential for efficiently keeping up with a topic you’re invested in.

• Be a fox, not a hedgehog. Make estimates considering everything you know, not just the one reason that seems most important to you.

Share your reasoning

• DAGGRE is designed to help the group make better predictions. The more information everyone has, the better.

• If you have information that others don’t seem to be considering, post it. That helps the market to make better decisions, and helps you: if people agree with your assessment you can take profits.

• DAGGRE now requires comments on >10% changes. Make your comments helpful.

Prioritize

• You only start with 1000 points. You can’t take a large position on every question.

• Your time is limited.• You won’t be an expert on everything.• Even if you are, some questions are just plain

uncertain.• Calibrate the size of your position to your

confidence.

Trade well

• If you want to maximize your points, you have to understand how the scoring system works, and trade accordingly.

• Good trading is not a substitute for making good predictions: doing both is the only way to climb the leaderboard very fast.

• There are good trading strategies for different kinds of questions, but first you have to identify them. That means making good predictions.

Understanding Scoring

• You move one estimate from 5% to 10%, one from 50% to 55%, and another from 90% to 95%. Which costs more? Which gets you more points if you’re right?

• You’re confident that both question A and B will resolve false. A is at 10% and B is at 20%. Which one(s) should you move? How much?

• When should you reverse an edit to take profits? (Hint: focus on liquidity.)

Trading Strategy 1: Patient Accumulation

• If question legitimately has a low (or high) probability, you want to profit off the occasional overconfident misprediction.

• Build up a cheap contrarian position when the estimate is really low (high), so you have points to reverse any large contrarian moves.

• Repeatedly move the estimate from 1% to 2%. If someone moves it up to 50%, move it back down to 10%.

• You’re not betting the consensus is wrong, just that someone else will come along and make an unwise edit. If not, you are only out a few points.

Trading Strategy 2: Genuine Uncertainty

• If a question is legitimately uncertain, you want to profit off edits that move the estimate too far in either direction.

• This strategy is simple: when the current estimate moves away from your last estimate, make a new estimate that splits the difference. If you estimated 50%, and someone puts it at 70%, move it back to 60%.

• The advantage of this strategy is that any position you take naturally gets liquidated over time, as edits move in both directions, and you partially reverse them.

Trading Strategy 3: Use Real Odds• Sometimes there’ll be questions (especially sports) where you can

find a quantitative probability of the event.• For example, http://www.playoffstatus.com/ and

http://www.oddschecker.com/ • Move the estimates closer to the actual probabilities.• Diversify: you’ll win some and lose some, but if you spread your

estimates across all the questions, you’ll likely gain points overall.• Update your estimates often as games are played and the odds

change.

Trading Strategy 4: Be First

• New questions come out at 50%, but usually don’t stay there. Edit early (in the correct direction) and gain points fast.

• Consider related questions, and make conditional edits.• Invest according to your confidence level. Treat others’

estimates as new information.• Once the probability is accurate, start gradually liquidating

your position. Always move the estimate back down to at least 98% (or up to 2%) if it gets all the way to 99% (1%).

• Consider switching to Patient Accumulation when you manage to fully liquidate.

Track your positions

• The strategies above require knowing when estimates change.

• Periodically check “My Positions” to see which ones have moved the most.

• Check the comments and your Google News Alerts for new information.

• Make a few edits each login. Check in frequently.

Follow-up

• Play the workshop market: it’s the fastest way to learn.

• Play the practice questions. I’ve offered free points to anyone demonstrating a good trading strategy there.

• Read Nate Silver’s “The Signal and the Noise”.• Come talk at breaks/lunch.• Connect with me on LinkedIn, Facebook, or

scottleibrand@gmail.com

Questions?

?