Frontier Models Know More Than They Say

Discrimination, reliability, and the failure of a numeric-anchoring hypothesis

1. Introduction

The prevailing frame for evaluating language models is capability: can the model do the task. For deployments where errors are expensive and rare events are costly, that frame is insufficient. What matters is whether the model's confidence tracks reality, so a downstream process can decide how much weight to put on the output. This is calibration, and it is the central competence of forecasting, medicine, aviation, and capital allocation.

The design question is narrow and answerable: when a frontier model states a probability, what does that number mean?

The benchmark was built to test a specific prior hypothesis, drawn from a year of running an autonomous agent in production and written down before any data was collected:

If true, this would be an important constraint on retrieval-augmented agent design: the standard pattern of loading every relevant number into the prompt would degrade judgment precisely on the tasks where judgment matters. The benchmark tests it by putting every question to every model twice, once with qualitative context alone and once with the same question plus quantitative context.

The result does not support the hypothesis, and section 10 shows the effect is statistically equivalent to zero. The unplanned finding, in sections 3 through 7, is the more interesting one. It supports the original claim, that calibration rather than capability is the binding constraint, by a route the hypothesis did not anticipate.

2. Method

Questions. 143 resolved binary markets from Manifold, filtered to at least 15 unique traders and screened for personal markets. Resolution dates span 2026-07-05 to 2026-07-20 against a run date of 2026-07-20, so every outcome landed within fifteen days of the run and well after every model's training cutoff. This is the primary lookahead control. The base rate is 36.36% YES, giving climatology a Brier score of 0.2314.

Conditions. Each question is rendered twice. Bare is the question plus up to 800 characters of the market's own description. Numeric is identical plus a block of quantitative context: participant count, trading volume, liquidity, market age, and a stated base rate. The resolution outcome is never included, and on the resolved track neither is the market's final community probability, which would hand the model the crowd's answer.

Models. Eight models across three providers, each run in two modes where possible: direct, with provider-native reasoning explicitly disabled, and thinking, with reasoning enabled at high effort. Fable 5 cannot disable reasoning and appears in thinking mode only, giving 15 model-mode entries. The system prompt asks for a single number between 0 and 1; replies are parsed permissively and clamped.

Coverage. 10,225 predictions in total across the bare condition, the ablation arms, and the held-out set of section 7. The scored set has zero nulls, so every entry is scored on the identical 143 questions and all comparisons are fully paired.

Inference. All uncertainty is non-parametric. Confidence intervals are bootstrap percentile intervals over questions. Paired comparisons use a paired permutation test on per-question squared error with condition labels exchanged within question, 20,000 permutations. Where seven models are tested for the same effect we report per-model p-values uncorrected and rest the aggregate claim on an across-model sign test, noting that at n = 7 the sign test can only reach significance at 7 of 7.

3. No model significantly beats the base rate

The primary result is negative. Everything in this section is scored on the bare condition: the question and its own description, with no quantitative context block and no stated base rate. None of the ablation arms of section 9 feeds any table here, so the leak described there does not touch these numbers.

Overconfidence throughout is mean confidence minus accuracy, where confidence is the probability assigned to whichever outcome the model called at a 0.5 threshold. Zero means perfectly calibrated; positive means stated confidence exceeds how often the model is right.

Climatology appears twice below. The obvious version, always predicting the set's own 36.36%, is itself a one-parameter model estimated on the test set, which makes it a slightly unfair opponent for a raw model and a fair one for a recalibrated model. The honest version predicts the out-of-sample 39% measured on disjoint markets. They differ by 0.0007, so nothing turns on the choice.

Ten entries score nominally better than climatology and five worse, but every interval contains the climatology value of 0.2314 and none is significantly better than it. At n = 143 the benchmark cannot distinguish any single frontier model from a forecaster that ignores the question entirely and repeats the base rate. One entry, Haiku 4.5 in direct mode, is significantly worse. Two score worse than saying 0.5 every time.

Overconfidence is positive for all 15 entries without exception. No model tested is underconfident.

4. But every model has real discrimination

AUC measures only whether a model ranks YES outcomes above NO outcomes. It is invariant to any monotone transform of the stated probabilities, so it sees knowledge while ignoring how confidently that knowledge is expressed.

5. The loss is reliability, not resolution

The Brier score decomposes as Brier = reliability − resolution + uncertainty, where uncertainty is a property of the question set alone (0.2314 here), resolution measures how far binned predictions depart from the base rate in the direction of truth, and reliability measures how far they depart from the observed frequency within each bin. Higher resolution is better; lower reliability is better.

The models are not scoring at climatology because they are ignorant. They are scoring at climatology because the information they extract is destroyed, almost exactly, by the miscalibration of the numbers they attach to it.

The ordering of the leaderboard is also mostly an ordering of reliability, not of resolution. Haiku 4.5 in direct mode has the worst Brier score by a wide margin, yet its resolution is only 37% below the median entry while its reliability is roughly three times the median. The leaderboard is, to first approximation, ranking honesty rather than knowledge.

6. One parameter recovers the signal

If the loss is reliability, a reliability correction should recover it. We fit the simplest possible correction, a linear shrinkage of stated probabilities toward the base rate:

p′ = b + a · (p − b)

with a single free parameter a per entry. This map is strictly monotone for a > 0, so it cannot change any model's ranking of questions and cannot add one bit of information. It only changes how far the model is willing to commit. To keep the estimate honest, a and b are refitted on the other 142 questions for each held-out question, so no entry is scored on a parameter fitted to its own outcome.

No entry has a above 0.64, including the best entry in the study, so every model tested should be compressing its stated probabilities toward the base rate by at least 36%, and the worst by 78%.

a is also monotone in raw Brier, but that is close to tautological rather than informative: reliability dominates the spread in Brier here, and a is precisely the parameter that corrects reliability, so the two are near-mechanically linked (rank correlation −0.97). Checking whether a tracks anything Brier does not, it does not: the partial correlation between a and AUC controlling for Brier is not positive. So we do not claim a is a better leaderboard metric than Brier. It is the same information in a more actionable form, and it is a property of a model crossed with a question set, not a constant of the model.

The practical reading is that a deployment which takes a frontier model's stated probability at face value is making an error it can remove with a hundred labelled examples and a one-dimensional fit.

7. The correction transfers to questions it never saw

The cross-validation above is a weak test of generalisation. With 143 questions every leave-one-out fold fits on 142 of them, so the parameter barely moves between folds. It answers "is this fitted to one question" (no) and not "does this survive questions it has never seen" (unknown). On its own that would not support a deployment recommendation.

So we ran the real test. We built a disjoint held-out set of 205 resolved markets, same population, same filters, same post-cutoff window, sharing no question with the scored 143 (the code asserts the intersection is empty). Every entry was re-elicited on all 205, a further 3,075 predictions. Both parameters of the shrinkage map, the slope a and the centre b, were fitted on the held-out set alone and applied unchanged to the 143. No information from the scored set touches the fit at any point. The reference is out-of-sample climatology at 0.2321.

The fitted parameter itself transfers only loosely. The a fitted on held-out data correlates with the in-set a at r = +0.71 but is systematically more aggressive, mean 0.44 against 0.52. The ordering of models is roughly preserved while the level is not, consistent with a being a property of a model crossed with a question set. What transfers reliably is the direction and rough magnitude of the correction, which is all a deployment needs.

The four entries that do not clear the baseline significantly are the four with the least discrimination. Shrinkage cannot manufacture signal; it can only stop a model destroying the signal it has. Haiku 4.5 direct is the clearest case: its held-out a is 0.07, meaning the honest fit is to discard almost everything it says and predict the base rate.

8. Reasoning mode does not reliably help

Enabling native reasoning helps four models and hurts three. The only significant effect is Haiku 4.5, the weakest and most overconfident entry in the study, where reasoning removes most of a very large deficit. For the six other models the effect is indistinguishable from zero and the direction is not consistent.

The natural interpretation is that reasoning repairs gross miscalibration but does not produce good calibration. Once a model is in the ordinary range of overconfidence, thinking longer about a forecasting question does not make the resulting number more honest. Reasoning mode is also the more expensive configuration by a wide margin, and on this task it buys nothing measurable for seven of the eight models.

9. Is it the model, or the format?

Every number above comes from one elicitation format: a bare decimal, requested in a system prompt, sampled once at default temperature. If overconfidence is partly an artefact of that format, the framing question has a different answer than it appears. We tested this on one model, Opus 4.8 in direct mode, same 143 questions, with two alternatives: the same prompt sampled five times and averaged, and a change of representation to odds against.

Sampling noise is negligible. Across five samples of the same prompt the within-question standard deviation averages 0.011, median 0.005, and 63 of 143 questions return five identical answers. Averaging correlates with the single sample at r = 0.994 and moves overconfidence by −0.016 (p = 0.20). For calibration purposes at this temperature, one sample is five samples. The second row is the useful reference: a fresh single sample in the same format shifts overconfidence by −0.015 from run-to-run variation alone, which is as large as the effect of averaging.

Changing the representation does not help either. Odds elicitation produces a visibly different distribution, correlating at r = 0.894 rather than 0.994 and compressing the spread slightly. But its overconfidence is +0.0731 against +0.0711, a difference of 0.002 (p = 0.95). Asking the same model for the same belief in a different notation moves individual answers around without touching the calibration.

10. The anchoring hypothesis, and a leak of our own

The hypothesis predicted a positive anchoring delta: overconfidence with quantitative context minus overconfidence without it.

A flaw in the first design

The benchmark's original numeric block ended with a line stating the base rate, and that figure was computed in-sample, from the resolutions of the same 143 questions being scored. The numeric condition therefore did not merely add quantitative context. It told the model the aggregate answer to the exact test set.

The leak-free 2 × 2

Two things changed. The base-rate line was pulled out of the numeric block so it could be varied independently, and the stated base rate was recomputed out-of-sample as the YES-share of 205 resolved markets from the same population but explicitly disjoint from the 143 being scored. That figure is 39%, against the scored set's 36.4%, so it is a genuine prior about the population rather than a restatement of the answer key. Crossing the two factors gives four arms, all direct mode, all on the identical 143 questions, each character-identical to the served prompt apart from the line under test.

That the effect survives the switch from an in-sample to an out-of-sample base rate is itself informative. The models are not exploiting a leak; they are using a stated prior about the population, which is what a forecaster should do.

Equivalence, not refutation

A null sign test is consistent with no effect but is not evidence of no effect. Two further cautions apply: the sign test at n = 7 is extremely blunt, since 7 of 7 is its only attainable significant outcome (p = 0.016) and 6 of 7 gives p = 0.125, so the design had exactly one shot at an aggregate directional result.

The stronger claim therefore needs an equivalence test. Taking the seven per-model main effects as the unit of analysis, the mean is +0.0059 with a 90% confidence interval of [−0.011, +0.023]. Two one-sided tests establish equivalence to zero for any smallest effect of interest above ±0.023: TOST gives p = 0.017 at ±0.030 and p = 0.036 at ±0.025, but p = 0.080 at ±0.020. We did not pre-register a smallest effect of interest, so this bound is descriptive rather than a passed pre-specified test.

Haiku 4.5 is the only model moving in the predicted direction at all (+0.088, p = 0.042), and it drives most of the variance behind that bound: excluding it, the mean effect is −0.002 and equivalence holds comfortably at ±0.020 (p = 0.005). One nominal p = 0.042 out of seven uncorrected tests is close to what chance produces, so this is a thread to pull rather than a finding. It is suggestive only because it points somewhere specific, namely that susceptibility to numeric anchoring might be a property of weak models rather than of numeric context, and Haiku 4.5 is the weakest entry here.

The fix. Ingestion now computes the base rate from markets explicitly disjoint from the scored set, records the reference count and the scored set's own YES-share in metadata so disjointness can be verified rather than trusted, and emits the base-rate line as its own arm instead of burying it inside the numeric block.

11. Discussion

Calibration is the bottleneck, but not for the reason predicted

The capability is present and measurable. Every entry ranks outcomes better than chance, several with AUC near 0.70, on questions that resolved after their training cutoffs, judged from a short prompt with no retrieval, no tools, and in direct mode no deliberation. That is a real forecasting signal.

The calibration is absent and expensive. All 15 entries are overconfident, reliability roughly cancels resolution, and the aggregate score lands at a baseline that ignores the question. The gap between what these models know and what their stated numbers convey is, on this set, worth the entire signal.

The practical implication is small and immediate

A pipeline that consumes a model's stated probability should not consume it raw. Shrinking toward a base rate with a single fitted parameter moved every entry from below climatology to above it. That is a change to one line of downstream code, requires no model access, no fine-tuning, and no additional inference cost, and it is worth more on this benchmark than the difference between the best and worst frontier model tested.

The corollary for evaluation is that a Brier leaderboard alone is close to uninformative. Section 3 would have supported the conclusion that these models cannot forecast. Sections 4 and 5 show that conclusion is wrong. Any calibration benchmark should report the decomposition, not the scalar.

What the shrinkage factor is, and is not

a is the fraction of a model's stated departure from the base rate that its discrimination actually justifies, on the question set it was fitted to. Both qualifications matter.

It is not an independent measure of model quality. Its rank correlation with Brier is −0.97, which is close to mechanical: reliability dominates the spread in Brier here, and a is precisely the parameter that corrects reliability. Controlling for Brier, it carries no additional signal about discrimination. So a is not a better leaderboard metric than Brier, and section 6 makes that explicit.

It is also not a constant of the model. The same entry yields different values on different sets, and section 7 shows how much: Haiku 4.5 in direct mode fits a = 0.22 on the scored set and a = 0.07 on the held-out set. Quoting either as "Haiku 4.5's shrinkage factor" would be wrong. Across all fifteen entries the two fits correlate at r = +0.71 with the held-out fit systematically more aggressive, so what is stable is the ordering and the rough magnitude, not the level.

What a is good for is the thing Brier cannot do: you can apply it. It is a number you multiply by, and section 7 establishes that a value fitted on one set still improves every entry on a set it never saw. That is the whole claim, and it is an operational one rather than an evaluative one.

12. Limitations

• Statistical power. At n = 143 every interval in section 3 contains climatology and most pairwise model differences are unresolvable. The paired within-model comparisons are far better powered than the between-model ones and should be weighted accordingly.
• A leak we introduced, and removed. The benchmark's original numeric arm stated the test set's in-sample base rate, making its scores uninterpretable. That arm is excluded from every result here and is superseded by the leak-free factorial of section 10, which uses an out-of-sample base rate. No number in this paper rests on the contaminated arm. The episode is a reminder that the most dangerous leaks are the ones an author writes deliberately, for what seem like good reasons.
• Single source and window. One platform, a fifteen-day resolution window, one season. The 36% base rate is a property of that window, and the topic mix skews to politics, sport, technology, and internet culture.
• Question quality. Manifold resolutions are set by market creators and are not uniformly rigorous. 126 of 143 prompts include scraped background text of uncontrolled informativeness, identical across conditions so it cannot explain the paired results, but inflating absolute discrimination to an unknown degree.
• Cutoffs are inferred, not verified. The lookahead control is the fifteen-day resolution window, which is a strong argument but not a documented per-model guarantee.
• Elicitation format is tested, but on one model and two alternatives. Section 9 shows the overconfidence here is not an artefact of single-sampling nor of the decimal representation. That closes the most obvious form of the objection, not the general one: interval elicitation, verbal likelihoods, higher temperatures and the other seven models are untested.
• Transfer is shown across questions, not across worlds. Section 7 fits on a genuinely disjoint set, so the correction is not merely cross-validated. But both sets are Manifold binary markets, filtered identically, resolving in the same month, with base rates of 39% and 36%. Nothing here establishes that an a fitted on prediction markets transfers to a different domain, a very different base rate, or a later time period, and the drop from 0.52 to 0.44 in mean fitted a between two sets this similar suggests the level is sensitive.