Forecaster Confidence Calibration

Overview

This pillar analyzes your personal prediction history to identify and quantify systematic biases like overconfidence or underconfidence. It's a meta-analytical tool to help you understand your own forecasting tendencies.

What It Does

It compares your stated probabilities on past markets with their actual outcomes. By bucketing your predictions, like all those you rated 70-80%, it calculates how often they actually came true. This process generates a personal calibration curve, visually revealing if your confidence matches reality.

Why It Matters

The single biggest leak in many forecasters' performance is their own cognitive bias. By making you aware of your specific tendencies, this pillar provides a data-driven way to adjust your future predictions, leading to better long-term accuracy and profitability.

How It Works

First, the pillar aggregates your entire history of resolved market predictions. It then groups these predictions into probability bins, for example 0-10%, 10-20%, and so on. Within each bin, it calculates the actual frequency of correct outcomes. Finally, it plots your stated probability against the actual outcome frequency to generate your calibration score and curve.

Methodology

Predictions are binned into deciles based on their forecast probability. For each bin, the mean forecast probability is plotted against the observed frequency of 'Yes' outcomes. The resulting calibration curve's deviation from a perfect 45-degree line (y=x) indicates miscalibration. A Brier score can be decomposed to measure this calibration component, while the slope of a best-fit line indicates systemic bias (slope < 1 suggests overconfidence).

Edge & Advantage

This pillar provides an edge by analyzing the most overlooked variable in any forecast: the forecaster themselves. It turns your own cognitive bias from a hidden liability into a measurable and correctable factor.

Key Indicators

• Calibration Curve Slope

  high

  Measures the direction and magnitude of miscalibration. A slope below 1.0 indicates overconfidence.

• Overconfidence Bias Score

  high

  A single metric quantifying the tendency to assign probabilities that are too extreme.

• Ideal vs. Actual Accuracy

  medium

  The gap between your predicted probabilities and the actual win rates for those predictions.

Data Sources

• User Prediction History

  The user's own historical record of resolved predictions and their associated probability estimates on the platform.

Example Questions This Pillar Answers

• → Am I systematically overconfident in my 90%+ predictions?
• → How should I adjust my confidence on political markets based on my past performance?
• → Is my underconfidence in longshot bets costing me potential gains?

Tags