Skip to content

Track record

Prediction track record

Before a site fix is published, Recognly can replay the questions it targets with and without the new page and register a prediction of the change. This page shows how those predictions compared with what happened once the fixes were live, failures and exclusions included.

Platform track record

Waiting for enough resolved predictions

Predicted in a controlled replay, not verified live.

Not enough resolved predictions yet: 0 of 30

Figures appear once at least 30 predictions from at least 5 organizations have resolved, so no single customer's results can be read off this page.

Registered, waiting and excluded counts appear with the figures, so no single customer's activity can be read from them.

The method

Before a fix to a brand's own site is published, the buyer questions it targets are replayed on one controlled model (Claude without web search) with the pages the engines actually retrieved: once with the page as it is and once with the draft, three paired samples each. The difference becomes a predicted change in the live rate per panel, with an 80% interval and the chance of a rise of at least 1 point. After publishing, the realised lift is measured against questions deliberately left alone (the holdout), and every prediction is scored.

Predictions describe a controlled replay; the realised lift is measured against questions we deliberately leave alone (the holdout).

How predictions are scored

  • Brier score: the squared gap between the predicted chance of a measurable rise (at least 1 point) and what happened. Lower is better; always saying 50% scores 0.25. Shown with a 95% bootstrap interval.
  • Skill against the base rate: above 0 means the predictions beat always predicting how often lifts were measurable.
  • Interval coverage: how often the 80% interval held the realised lift; close to 80% is well calibrated.
  • Reliability: predictions in five bins by their chance, against the share that rose, with 95% Wilson intervals.

Each panel is shown on its own; panels are never combined. Each method version is reported on its own.

What is counted

Every registered prediction is counted. Its record is fixed when it is made and cannot be changed or deleted. Predictions are excluded only by rules fixed in advance: another fix or reply touched the same questions in the measurement windows, the fix was not published within 60 days, the draft changed after the prediction (a fix has one live prediction at a time: predicting the edited text replaces the older prediction, which is kept and counted as superseded, and the same text is never predicted twice), too few answers to measure, or the holdout questions changed. Each exclusion is counted above. Organizations can leave their predictions out of these pooled figures; sample and demo data never appear here.

Nothing here is a promise about how any AI engine will answer.