Personal Brier Score Tracker

Overview

This pillar provides a rigorous accounting of your personal prediction accuracy using the Brier score. It moves beyond simple win/loss records to offer a true measure of your forecasting skill, helping you identify biases and improve over time.

What It Does

The pillar calculates a Brier score for each of your resolved predictions, which measures the accuracy of probabilistic forecasts. It then aggregates these scores to provide a rolling average and long-term performance trend. This allows you to objectively track if your calibration and accuracy are genuinely improving.

Why It Matters

Profit and loss can be swayed by luck or risk management, but the Brier score directly measures the quality of your probability estimates. Tracking this score provides an objective, data-driven assessment of your forecasting ability, which is the foundation of long-term success.

How It Works

First, you log your predictions, including the market, your assigned probability, and the final outcome once it resolves. The system then calculates the Brier score for that single prediction. This score is then used to update your overall and rolling performance metrics, visualizing your accuracy trend line.

Methodology

The core calculation is the Brier score: (forecast probability - actual outcome)^2, where the outcome is 1 for 'Yes' and 0 for 'No'. The pillar maintains a rolling average of this score over the last 50 resolved predictions to reflect current skill. It also plots a calibration curve to diagnose overconfidence or underconfidence.

Edge & Advantage

This tool replaces subjective feelings about your performance with hard data, enabling systematic identification of personal biases for targeted improvement.

Key Indicators

• Rolling Brier Score

  high

  The average Brier score over your last N predictions, indicating recent forecasting accuracy. A lower score is better.

• Calibration Curve

  high

  A plot of your predicted probabilities against the actual frequency of outcomes, revealing systematic over or under-confidence.

• Resolution Accuracy

  medium

  The simple win/loss percentage of your predictions. Less nuanced than the Brier score but useful for a quick overview.

Data Sources

• User's Prediction History

  A log of the user's own past predictions, including the market, their assigned probability, and the final resolution.

Example Questions This Pillar Answers

• → Is my forecasting accuracy improving over the last 3 months?
• → Am I systematically overconfident when I predict probabilities above 80%?
• → Do I perform better on political markets compared to crypto markets?

Tags