T&R
Truth & Reconciliation · a forecasting research project

Every forecast, sealed before the answer exists.

Each morning, T&R asks several AI models how likely real events are to happen: an inflation print, a Treasury yield, a grid's peak load. It locks those numbers away, waits for the events to settle, and only then checks who was right.

What it is

A daily, auditable test of machine judgment

Truth is the forecast. Reconciliation is what happens to it when the event resolves.

Plenty of people claim that AI models can forecast. Very few of those claims survive the obvious questions. Were the forecasts written down before the answer was known? Were the bad ones kept? Did someone pick the flattering subset afterwards? T&R is built so those questions have boring answers.

The idea comes from Superforecasting: a crowd of forecasters beats most of its members, as long as they don't all think alike. T&R applies that to AI models, treating each one as a separate forecaster and using them for diversification. The research question is how much diversity that actually buys: how different the models really are from one another, how much recent news changes their answers, and what to improve between rounds of testing.

The questions come from listed event markets on Kalshi and Polymarket, because those come with precise settlement rules and a public record of how each one resolved. Everything the models are shown, and everything they answer, is stored and fingerprinted, so any forecast can be traced back to the exact inputs that produced it.

It is a research project. It does not trade, take money, or sell signals.

A day in the ledger

One round, every day, on a fixed clock

Times are UTC. The evidence cutoff is fixed, so a forecaster only ever sees what was public at 11:45 that day.

  1. Morning

    Pick the questions

    A fixed rule selects open questions from the venues: enough time left to be interesting, clear settlement rules, no question where anyone involved could influence the outcome.

  2. 11:45

    Freeze the evidence

    Rules text, recent official data and news headlines are gathered and frozen at the cutoff. Nothing published after 11:45 can reach a forecaster.

  3. 12:00

    Forecast

    Each forecaster answers every question with a probability and a short rationale. Failures, timeouts and refusals are logged as failures, not dropped.

  4. Same day

    Seal

    The round is sealed: every input and output is content-hashed into an append-only record. A sealed forecast cannot be edited.

  5. Days to weeks

    Settle

    When a question settles, the outcome is captured from the venue and checked independently. The first verified outcome is final; there is no rewriting history.

  6. After settlement

    Score

    Only then is each forecast scored, using the Brier score: the squared distance between the probability and what happened. Lower is better; always saying 50% scores 0.25.

The forecasters

Three models, each with a twin

Each model provider runs two seats. The news seat sees the settlement rules plus current evidence. Its rules twin sees only the rules. Comparing the two answers one clean question: how much does the news actually change what a model thinks?

A seventh seat is a dumb reference that ignores the question entirely. Any forecaster worth listening to has to beat it.

SeatModel familySees
C · newsAnthropic ClaudeRules, official data, headlines
C · rulesAnthropic ClaudeRules only
G · newsGoogle GeminiRules, official data, headlines
G · rulesGoogle GeminiRules only
O · newsOpenAI GPTRules, official data, headlines
O · rulesOpenAI GPTRules only
ReferenceNone (a fixed baseline)Nothing

The six model seats are also averaged into one ensemble forecast, which is recorded alongside them.

Findings from the first month

What the models do, before anyone knows who was right

These come from the first 22 daily rounds, 27 Aug to 27 Sep 2026: 859 recorded forecasts on 158 questions, mostly economics and financial data.

Every finding here describes how the forecasters behave. None of them says whether the forecasts were accurate. Accuracy is deliberately not published while the project is under way; see the ground rules below.

1.4 independent views

Six seats, roughly one and a half opinions

Across questions the six model seats move together: the average correlation between any two is 0.81. Treated as a panel, six seats carry about as much independent information as 1.4 forecasters would. Adding more of the same kind of model buys less than the headcount suggests.

Pairwise correlation of forecast probabilities on questions both seats answered. Darker means more alike.

0.94 vs 0.71

The model matters more than the news

Each model agrees more with its own rules-only twin than with any other provider given the same news. Claude-with-news correlates 0.94 with Claude-without-news, but only 0.71 with OpenAI-with-news. Which model you ask moves the answer more than what you show it.

with its own rules-only twinwith the other two news seats (average)
5 points

News nudges, it doesn't steer

On the same question, a model's news seat lands on average about five percentage points away from its rules-only twin: 4.7 for Claude, 5.0 for OpenAI, 5.3 for Gemini. The shifts point both ways and nearly cancel, so news is not making the models systematically more or less optimistic.

Only about one news-fed forecast in five (20%) cites a headline in its rationale. Most of the time the model reasons from the rules and its own background knowledge, and the headlines sit unused.

News in this period came from a public headline feed and was split between providers, so different seats saw different stories. The feed has since changed, and these numbers describe the earlier setup only.

53% on round numbers

Models like a tidy number

More than half of all forecasts land exactly on a multiple of five percent, and 50% is by far the most common single answer. Real uncertainty is rarely that tidy, so this looks like anchoring. It varies a lot by provider:

Share of each seat's forecasts that are exactly 50%. The OpenAI seats say "even odds" about one time in three, and they are the least extreme overall: only about 5% of their forecasts fall below 5% or above 95%, against about 23% for Claude.

7.7% of attempts failed

Failures are data too

72 of 931 forecast attempts produced no usable answer: the model tool crashed, the provider rate-limited us, or the output was cut off. They are kept in the record as failures. If the hardest questions fail most often, dropping failures would make every seat look better than it is.

About these numbers: one month of data, with many questions grouped into families that settle on the same underlying figure (nine strikes on one inflation print are one event, not nine). 158 questions come from about 139 such families. One of the model seats also changed model version partway through. Read everything above as a description of this period, not a law about these models.

Ground rules

How we keep ourselves honest

Forecasting research is easy to fool yourself with. These rules were written before the data they govern.

Every record behind one door. It opens; nothing goes back out changed.
The archive. Every input, every answer, every failure, kept where nobody can edit it.
  • The test is registered in advance.What would count as an edge, on which rounds, against which baseline, was written down and frozen on 12 October 2026. The test window runs 1 November to 31 December 2026. A change to the forecasters during the window ends the window early. Nobody gets to choose the cut.
  • Scores are under embargo.Accuracy from rounds dated 28 September to 31 December 2026 is not looked at, by anyone, until the analysis runs. Peeking at a running score is how tests get tuned without anyone meaning to.
  • Events are counted, not strikes.Questions that settle on the same number are averaged into one unit before anything is scored, so a ladder of related questions can't fake precision.
  • The first outcome is the outcome.A verified settlement is permanent. There is no quiet correction afterwards.
  • Every input is kept.Each forecast is stored with the exact text the model was shown and fingerprinted, so it can be checked later by anyone holding the record.
  • No inside information.T&R does not forecast anything where anyone involved could influence the result or knows something the public doesn't.