← henrywhelan.com
Interactive · calibration benchmark

One parameter

Frontier models rank the future well and state it badly. Every dot below is a real prediction on a real question. Drag one number and watch the knowledge come back.

The map is p′ = b + a·(p − b): shrink each stated probability toward a centre b by a factor a. It is monotone, so it cannot reorder a model's questions or add information. It only changes how far the model commits. a = 1 is the model's own numbers. The held-out fit uses each entry's a fitted on 205 disjoint questions the models never saw, applied unchanged here. Companion to the paper; live table on the leaderboard.

shrinkage a · shared by all entries · b = held-out base rate a = 1.00raw predictions
0 · climatology0.51 · as stated1.2 · bolder
—below climatology
—significantly better
—mean Brier gain
loading 2,145 predictions…