Calibration and Brier Scores: How to Judge Any Forecaster's Track Record

Anyone can say they are "usually right." The interesting question is: right about what, at what confidence, and over how many calls? This post gives you the two tools that separate real forecasting skill from marketing — calibration and the Brier score — plus the honesty check most people skip: sample size.

Why hit rate lies to you

The first number every touting account shows is a hit rate: "80% accuracy," "6 out of 6 today." It feels informative. It is almost useless on its own, for one reason: hit rate is trivially gameable by only making easy calls.

If I only ever forecast events that are 95% likely, I will be "right" about 95% of the time and look like a genius. But I have told you nothing you did not already know, and I have taken zero real risk with my judgment. Conversely, a forecaster who takes on genuinely uncertain 60/40 calls might post a lower hit rate while being far more useful, because they are adding information exactly where it is scarce.

Hit rate conflates two different things: how hard the questions were, and how good the answers were. To pull them apart, you need a metric that grades probabilities, not just yes/no outcomes.

The Brier score, in one minute

In 1950, meteorologist Glenn Brier proposed a way to score probabilistic forecasts (Brier, "Verification of Forecasts Expressed in Terms of Probability," Monthly Weather Review). The idea is simple: for each forecast, take the probability you assigned to what actually happened, and measure how far off you were, squared.

• You say 90% and it happens: your error is (1 − 0.90) = 0.10, squared = 0.01. Small penalty.
• You say 90% and it does not happen: your error is (0 − 0.90) = −0.90, squared = 0.81. Large penalty.

Average that squared error across all your forecasts and you get the Brier score. It ranges from 0 (perfect) to 1 (perfectly, confidently wrong). A useful reference point: always guessing 50% on binary events gives a Brier score of 0.25. So anything meaningfully below 0.25 is doing better than a coin-flipping shrug, and scores in the low single-digit-hundredths reflect confident forecasts that were also usually correct.

For context, on our own public record the Brier score is 0.058. We will come back to why you should still be a little skeptical of that number.

Calibration: were you as confident as you should have been?

The Brier score can be decomposed (Murphy, 1973) into components, the most intuitive of which is calibration — sometimes called reliability. Calibration asks a beautifully concrete question:

Of all the times you said "70% likely," did the thing actually happen about 70% of the time?

A perfectly calibrated forecaster's stated confidence matches reality across the board: their 90%s come true ~90% of the time, their 60%s ~60% of the time, and so on. You can literally plot this — predicted probability on one axis, observed frequency on the other — and a calibrated forecaster's dots hug the diagonal.

Calibration is what "hit rate" can never show you. A forecaster can have a great hit rate and terrible calibration (they say 99% when they mean 80%, and it bites them on the tail). Calibration rewards knowing what you don't know. It is the humility metric.

Philip Tetlock's Superforecasting (2015) made this famous: the best forecasters were not necessarily the smartest people in the room, but the ones whose confidence tracked reality — who updated in small steps and rarely said 95% about something that was really 70%.

The number nobody wants to say out loud: sample size

Here is where most track records fall apart, and where honest ones distinguish themselves. A Brier score or hit rate on a small number of calls is mostly noise.

Our public accuracy figure is 94.9% on n=59 settled calls. We say "n=59" out loud, every single time, on purpose. Fifty-nine is a small sample. With numbers that small, a couple of unlucky settlements would swing the percentage meaningfully. The honest interpretation is not "this proves we have edge" — it is "this is consistent with edge, and you should keep watching as the denominator grows."

A few rules of thumb for reading any track record:

1. Always ask for the denominator. "6 for 6 today" is meaningless without the season total. A real record shows the misses.
2. Small samples are compatible with luck. A 90% hit rate on 10 calls could easily be a coin that landed well. The same rate on 500 calls is a different conversation.
3. Confidence intervals widen fast at low n. With ~59 calls, the true accuracy could plausibly sit several points either side of the headline. Anyone who quotes a small-sample number as if it were precision is either naive or selling.
4. Look for the trend, not the snapshot. Does calibration hold up as more calls settle, or does the score drift toward the coin flip?

The tell of an honest operator is that they volunteer the sample size and its limitations. The tell of a shill is a big percentage with no denominator and no losses shown.

How to actually audit a track record

Put it together into a checklist you can apply to anyone — including us:

• Timestamps before resolution. A call only counts if it was public before the market settled. Screenshots taken afterward are worthless.
• Full sample, losers included. No cherry-picking. The denominator and the misses must both be visible.
• Calibration, not just hit rate. Ideally a reliability plot; at minimum, a Brier score alongside the accuracy.
• An auditable benchmark. The claim should be measured against something you can independently verify. We benchmark against a public, on-chain account, @car (+$1.29M over 582 days, ~93%/yr) — a number anyone can pull from the blockchain rather than take on faith.
• Stated small-sample honesty. If the sample is small, the operator should say so first, before you have to ask.

We built our public dashboard specifically so you can be adversarial with it. Every settled call, the accuracy, the Brier score, the small-sample caveat — all in one place, so you can stress-test the claim before trusting it. That is also the spirit of our companion piece, A Skeptic's Checklist for Paid Trading Signals.

The bottom line

Judge forecasters the way you would judge a weather service, not a fortune teller. Hit rate tells you how easy the questions were. Calibration tells you whether the confidence was honest. The Brier score rolls both into one number. And sample size tells you how much to trust any of it. Master those four ideas and you will never again be impressed by a screenshot of "6 for 6."

New to the category? Start with What Is a Prediction Market?. Curious why disciplined selection can produce a record like this in the first place? The academic backdrop is in The Favorite-Longshot Bias.

See a live, timestamped record for yourself. We are in an early testing period and giving free memberships to early testers — every call is logged before it settles, so you can watch calibration accumulate in real time. Join our Discord and DM the founder (or open a ticket) to claim: https://discord.gg/C6hX9w94Ej. Or just audit the public dashboard — no signup required.

Independent research service. Not affiliated with Polymarket. Illustrative of past results, not a promise. Not investment advice.