Chapter 7 · Keeping score
How YOLO knows it is any good
Any platform can say its models improve. What can actually be checked is the record-keeping underneath: decisions recorded before their outcomes exist, refusals recorded exactly like actions, confidence checked against what really happened, and the criteria for switching a model off written down before it was switched on. This chapter walks that loop, and links to the live record for YOLO's own AI expert — losses shown at the same weight as wins.
The loop that stops us grading our own homework.
Six steps, none of them clever. Each one is a rule about what gets written down and when — which is the only reason any claim about getting better can be checked at all. Open a step to see what it is protecting against.
Every specialist records what it decided at the moment it decided it — one row, one shape, the same for all ten of them.
- Why it mattersRefusals are recorded too. A system that only writes down the trades it took will always look better than it was, because its worst calls left no trace.
- Why it mattersThe row is written before the outcome exists, so nothing about it can be adjusted once the answer is known.
- Why it mattersOne shape for every specialist means they can be compared with each other rather than each grading its own homework.
When the position closes or the idea expires, the result is written back against that same decision.
- Why it mattersIt is keyed to the decision, so a claim and its result cannot drift apart or be re-paired later into a flattering story.
- Why it mattersLosses are attached exactly the way wins are, with no separate path and no discretion about which ones get recorded.
A specialist that says "seven in ten" should be right about seven times in ten. That is checkable, and it is checked.
- Why it mattersThe check hands back the evidence behind a number — the sample it used and the window it covered — or it hands back nothing at all. Never a bare figure with no provenance.
- Why it mattersToo small a sample yields no claim rather than a confident-looking one. Silence is a valid output here as well.
Every served number carries the identity of the thing that produced it.
- Why it mattersWithout a version, "the model has improved" is unfalsifiable — you cannot tell an improvement from a different model answering.
- Why it mattersIt also means a bad run can be traced to what was actually running at the time, rather than to what is running now.
Each model has an owner, a stated hypothesis, and the conditions for promoting it — and for switching it off — recorded in advance.
- Why it mattersWritten first, so nobody can move the goalposts after seeing the results. This is the same standard applied to human experts on the platform, deliberately.
- Why it mattersA model that drifts away from what it promised is demoted automatically. Getting it back to public requires a person, not a quiet recovery.
A new way of deciding runs alongside the current one first, recording what it would have done without doing it.
- Why it mattersIt earns its place on its own record, over the same period, against the same decisions.
- Why it mattersNothing you hold is the test bed for an idea that has not yet been measured.
What a confidence check looks like
For each band of stated confidence: what was claimed, against what actually happened. The gap is the whole point — a system whose bars matched perfectly would be one nobody had checked honestly.
- Low confidenceSaid30%Happened34%
- ModerateSaid50%Happened46%
- HighSaid70%Happened64%
- Very highSaid85%Happened81%
Worked illustration. These four bands are a made-up example so the shape of the check is visible. The real, live record for YOLO’s AI expert — the actual bands, the actual sample sizes, the actual window, with losses at the same weight as wins — is published at How accurate is our AI expert? It is a record of honest measurement, not a performance claim, and it makes no promise about what happens next.
Why the boring half is the important half
Every trading platform says its models improve. The sentence costs nothing, because almost nothing about it can be checked from the outside. What can be checked is the record-keeping underneath it — and that is where a system either has integrity or does not.
- Decisions are written down before outcomes exist. That ordering is the whole guarantee. A record written afterwards is a story.
- Refusals count. The times a specialist declined to act are recorded exactly like the times it acted. Leave those out and every system looks brilliant.
- The rules for switching a model off were written before it was switched on. Deciding afterwards what counts as failure is not a standard, it is a negotiation.
- A number without its evidence is not served. If the sample is too thin to support a claim, the honest output is no claim.
None of this is a forecast. Measuring the past carefully is how a system stops fooling itself; it is not a way of knowing what happens next. YOLO makes no promise about future results — it makes a promise about how honestly it keeps score.