← All research

How our AI forecaster audits itself every Saturday

Short answer: the method has a version number, and every Saturday an AI agent audits it. It looks at the week's misses, tests new ideas on years of price history, and lets a second agent try to break each result. A change ships only if it survives. Each version is scored on its own on the scoreboard, so a change can't hide in an average.

What this does not promise is that each version is better. We can audit the method every week. We can't make the market predictable.

What a version is

A version is the full set of rules the forecaster follows before it makes a call: what it reads, which statistics it is shown, when it must stay at 50%, how sure it is allowed to be. Version 1 read the morning news and nothing else. Version 4 added the day's economic calendar and a minimum amount of searching. Version 6 scales down the expected size of the US session on US holidays.

Every forecast records the version it was made with. A new version replaces the old one from a stated start date, and the old version's record is closed and kept. The scoreboard's headline numbers combine all versions, and each version is also scored on its own, so a change can't hide in the overall number: when the rules change, that version's own count starts again.

The Saturday audit

The audit is a Claude skill: a written procedure that an AI agent follows from start to finish. It runs once a week for each "nest" of forecasters (currencies and stocks), and each step is done by a separate agent:

  1. Score the week. Every call is matched with what the price did, and every miss gets a short label for why it missed: no clear driver, news that came out during the session, a data surprise, a move that had already happened before the session opened.
  2. Scout. A research agent searches papers and published studies for ideas that target the most common misses, and drops ideas that were already tested and rejected.
  3. Backtest. For up to four ideas, a backtesting agent writes down the pass bar before it looks at the data, then tests the idea on years of history it has to treat as unseen.
  4. Skeptic. A separate agent, whose only job is to find the flaw, reviews each result: did the test peek at the future, try too many variants, rest on too few cases, work only in one period, or move the goalposts?
  5. Ship or don't. Only ideas that pass both the bar and the skeptic go into the next version: at most one new version per week. The change is written into the forecaster's rules and merged automatically. If nothing passes, the method stays as it is.
  6. Write it down. The week's report, the ideas list, and one public line in the change log, whether or not anything changed.

Most weeks most ideas fail. In the first two audits, eight ideas were tested and two passed. One of those became a rule that removed something: statistical "edge" flags no longer tilt a call, because the skeptic showed they only echoed each pair's long-run drift.

What a person decides

We look after the mechanism, not the individual decisions it makes. That means the rules of the audit, the pass bars, the skeptic's checklist and the data it may use. We don't pick by hand which idea ships.

A person decides only two things directly. The first is lessons: short rules drawn from repeated mistakes, such as "with no dated news, stay at 50%". The audit can propose a lesson, but only a person can switch one on, and each one appears in the change log as "approved by a person". The second is pausing a forecaster that keeps failing. The early versions 1 to 4, made by hand while the test was being set up, are marked as such in the change log too.

How well this works is on the scoreboard, version by version, rather than in our description of it.

How to read the version numbers

  • Every call shows its version (v6) under its date on the scoreboard.
  • The versions table shows each version's dates, its number of calls and how many were right.
  • Below about 50 calls, a version's hit rate is mostly luck. With 30 calls, the 95% range around a hit rate is roughly ±18 points, so a version at 60% and one at 45% can't be told apart. The numbers are there so nothing is hidden, not as a ranking.
  • "No new version" is a result. It means ideas were tested and none survived, which is what you would expect from a market that is hard to predict.

Why bother if it doesn't get better?

Because the alternative is worse. A forecaster that is never checked drifts. One that is changed whenever someone has a hunch is overfit to last week. Changing only on tested evidence, recording every change, and scoring every version separately is the slowest way to improve, and the only way that doesn't fool us. If an edge ever shows up, it will show up there, in a version with enough calls to mean something. If none does, the record shows that too.

About the data: The weekly audit reports and the change log live next to the forecasts in our data repository. Ideas are tested on 15 years of hourly EUR/USD, EUR/CHF and USD/CHF prices (Dukascopy via forexsb) and 10+ years of daily stock prices.

Educational content, not financial advice. Past results don't guarantee future ones.