Why Offline ML Evaluation Fails

A model that fired about twice a day in offline evaluation fired zero times in four days of live operation. The model weights were the same. The data fields matched. The difference was that offline evaluation supplied the whole session, while live operation supplied only the observations available at the decision; when the offline session was truncated to each decision, none of the model's 25 apparent decisions survived. Walk-forward validation had not detected it, because the leak was inside each evaluated session. That case is one of eleven documented failures on one platform in ten months, of look-ahead leakage, selection bias and offline/live evaluation mismatch, each of which survived the validation procedures originally applied to its experiment. This is the record of the eleven, the tests that expose each class, the evaluation policy that now stands before any number is quoted, and the causality gate that automates part of it.

Key findings

• None of the eleven documented failures originated in a model's forward pass. The convolutions were causal and the attention masks were square; the problems arose in upstream data preparation, population selection, label generation, reference data, or the offline evaluation itself (a census at the wrong cadence, fills from prices the decision never saw).
• In the documented probe, the leaking action score increased as more future observations were supplied. On one session it was 0.001 with the tape cut at the decision, 0.011 with an hour of future, and 0.899 with the whole day; even an additional hour of observations did not reproduce the full-session score.
• Test the continuous quantity, never the thresholded decision. The harness's own self-test passed a whole-session-mean scorer when its output was thresholded and failed it when scored continuously.
• A scorer that never commits has no verdict, not a pass. The harness refuses to certify a checkpoint that acts on nothing in the scanned sessions.
• Prefix tests expose computations that depend on future observations within a sequence. Reference-data audits, population checks and live-path replay are needed for the failures truncation cannot reveal, and six of the eleven were of those kinds.
• The evaluation policy that followed is five checks, with the audits and the replay as separate boundaries beside them; the causality gate is the one command that automates the truncation, mutation and live-probe part of it. The gate is mandatory before a production checkpoint changes; the evaluation policy requires it before any research number from a model or census is quoted, recommended or deployed. 81 of the platform's 154 test files are tagged for the gate, a count of tests rather than a measure of coverage.

1. The claim

Ordinary offline evaluation can reproduce a look-ahead leak without detecting it. Walk-forward validation moves the training window forward in time; it does not change the evaluation's view of each session, which is always the whole day, so a feature computed from the whole day scores in research exactly as it will not in production. Interleaved halves and multiple seeds test stability, not causality. Two claims should be kept apart. The observed one: these eleven problems survived the validation procedures originally applied to their experiments, which were not the same procedure in every case. The general one: conventional walk-forward metrics, seed checks and ordinary batch scoring do not by themselves establish causal correctness or serving parity. Prefix tests expose computations that depend on future observations within a sequence: hand the scorer the session as it stood at the decision and ask whether the decision changes. Historical reference-data audits, population checks and live-path replay are needed for the failures that truncation cannot reveal, because in those cases the future is already inside the input before anything is truncated, or the mismatch is between two execution paths rather than inside one.

"Causal by construction" is the phrase under which all eleven leaks shipped, and the phrase was true of the parts anyone thought to check. The sequence models used causal convolutions and square subsequent masks, and the mutation tests found no future dependence in the fine features on the tested cases. The leaks lived between the components whose causal properties had been checked. Verifying the model architecture alone did not establish that the surrounding data and evaluation pipelines were causal.

2. The eleven cases: leakage, selection bias and evaluation mismatch

Eleven recorded cases, not eleven independent root causes and not mutually exclusive mechanisms: the fifth row is downstream contamination of the fourth, and the tenth row holds two reference-data problems found a week apart. By the class column, four cases involved future-dependent computations, one involved labels contaminated by an earlier leak, two involved offline/live execution mismatches, two involved outcome-selected populations, and two involved historically unavailable reference data.

One more belongs beside them. The harness's self-test, on the day the harness was written, passed a deliberately leaking scorer when its output was thresholded and failed it when scored continuously; the rule that followed is in section 5.

3. One case end to end: the score that needed the whole day

A sequence model had been trained to decide, every ten seconds, whether to act. Its research census showed it firing about twice a day with a strong record, and when it went live it fired zero times in four days. Everything ordinary was ruled out: the model and its forward pass produced the census's values when fed a cached session, the session fields matched, the previous close and float matched, the tape density matched, and the live flow gate was open on the columns where the census fired its best trades.

What was left was the shape of the two evaluations. Live scores a growing prefix of the session; the census scores the whole day at once. Truncating each of the census's firing columns to the session as it stood at that column, 0 of 25 survived, and the action score fell from between 0.52 and 0.99 to between 0.000 and 0.42. Feeding the model progressively more of the future beyond the decision column raised the value on every session probed, which is the figure above: on one session 0.001 with no future, 0.011 with an hour of it, 0.899 with the whole day, so even an hour of observations did not reproduce the full-session score. The partial-column explanation was dead at ten seconds, because at that point the decision column was complete and the score was still near zero.

A stage-by-stage mutation found where the future entered. The fine features did not move when every bar after the column was corrupted; the session grid moved; the codebook's token moved; the belief and action heads moved. The leak was upstream of the convolution stack, in the grid and the codebook: the code stream had been centred by the mean over all 540 tokens of the session, a history marker at a column could exist or not depending on tape after it, and that marker read the first print of the next column, one second of future sitting on the candle boundary. Only the third was subtle. The first two had been invisible because everything around them was correctly causal, so nobody had checked the parts in between.

The blast radius was every result built on that grid: the codebook, the model's belief and action outputs, the labels its belief had produced, the pyramid censuses and a second lane's gate. Live firing zero had been correct behaviour throughout. Four days later causality became a hard requirement in the operating manual, with the four leaks that caused it written at the top of the section, and a single command that must pass before a production checkpoint changes.

4. Why conventional offline evaluation missed them

Each row is reproduced, not detected, by ordinary batch scoring, because in every row the evaluation and the defect agree with each other. Future-dependent computations can be exposed by comparing full-session outputs against prefix-only outputs at appropriately selected decision points, which is what the harness does. The reference-data and population rows do not: the future is already inside the input, so the same wrong value comes back at every cutoff, and they need a capture-date audit or a population check instead. The cadence and fill rows are mismatches between two execution paths, and only a replay through the serving path shows them.

5. The harness

The harness takes any scorer that maps a session, a dictionary of per-bar arrays plus scalars, to one decision value per ten-second column. It discovers, per scorer, the columns where that scorer actually commits, so that a retrained checkpoint cannot quietly measure nothing, and it judges only columns whose prefix holds enough bars to be scored at all. Then it runs two checks:

• Truncation. The decision at column j computed on the session cut at j must equal the decision at j computed with the whole day.
• Mutation. Corrupt every bar after a column, prices shuffled and volumes rescaled, rebuild, and require everything at and before that column to be bit-identical to the clean build. Mutation perturbs future observations while preserving the sequence structure; it complements truncation by detecting value-dependent future reads that a truncation-specific branch in the code could otherwise mask. A read of the sequence's length, by contrast, is caught by truncation and not by mutation, since mutation leaves the length alone.

Two rules make the checks mean something. First, score the continuous quantity the decision thresholds, the probability, the logit, the predicted return, never the 0/1 decision. Second, no verdict is not a pass: a scorer that never commits in the scanned sessions is uncertified, and the gate exits with a code of its own for that case rather than a green one.

The causality gate is one command with three stages: the mutation tests, every test in the suite tagged causality, and the acceptance probe on the live configuration through the live loader, which checks that each decision the deployed checkpoint actually commits to is unchanged when the session is truncated to that instant. Exit code zero means the gate passed. Anything else means do not quote and do not deploy. The gate is enforced before the production checkpoints change; before a research result is reported it is required by the evaluation policy, a procedural rule rather than a technical one, which is the distinction this article is about. Every new lane, model, feature, label or census ships with a truncation test tagged for it before its first number is quoted.

Why the continuous score, not the decision

Imagine a decision bar of 0.5. A leaking score changes from 0.10 with the prefix to 0.40 with the whole day. Both values are below the bar, so a test that compares only the decision sees no difference and passes; a test that compares the score sees 0.10 against 0.40 and fails. The harness's own self-test found exactly this on the day it was written: a scorer that centred on the whole-session mean passed when its output was thresholded and failed when scored continuously. The codebook leak above shows a much larger discrepancy, from 0.001 to 0.899. A binary-decision comparison would detect that one at a bar of 0.5, but continuous-score testing remains necessary because smaller leaks change the score without changing the decision. The rule is to test the quantity the decision is made from, even when the system's output is binary.

6. The evaluation policy that followed

Before any number from a model or census is quoted, recommended or deployed:

1. Truncation. The decision at column j on the session cut at j equals the decision on the whole day. A scorer that never commits has no verdict, not a pass.
2. Live-path replay. Score real recent days through the serving class, never a research loader.
3. Gate parity. A census applies the live gate the way live does, per column, with that lane's thresholds. Latching at the first opening had overstated one lane's trades twofold; another lane's thresholds had made 80% of its census births unreachable.
4. Labels are only as causal as what made them. A model-managed label inherits that model's look-ahead.
5. Artifacts carry their causality. A manifest sits beside every codebook and grid cache, and loaders refuse a mismatch.

The five checks are not one boundary. They enforce model-output causality (check 1), offline/live execution parity (checks 2 and 3), label lineage (check 4) and artifact integrity (check 5), and the causality gate command automates the first of these and the replay probe; the others are enforced by the loaders, the census code and review. "Evaluation policy" below means the whole set; "causality gate" means only what the command runs. Two classes of failure sit upstream of anything the five checks can cut, and they got rules of their own after the September audits. A number joined from a reference table must have been knowable on the day: check capture dates, not column names. And a "pre-known constant" derived from same-day data is not pre-known. Per-second truncation of a model's inputs cannot see either; the upstream builders are tested by swapping the anchor for the true value on the same sessions and watching the decisions. In October the same discipline moved into the platform's feature store, which certifies every feature group by truncation, mutation and a membership cut before it is written; that is its own article, and this one is the record of the failures that made it necessary.

7. What it changed

The September audits replayed every premarket lane on the full population with the true previous close and each lane's own live gate. Every one that had looked profitable in research lost on honest inputs, and the platform's research archive records 32 threads closed or refuted, each with its numbers, checked before any new proposal. That is a negative result and it is the point: a system built to be able to say no said no. Live firing zero had been the system working. The lesson for the platform was not a better model but a harness any scorer must pass, a replay through the serving class, and a store that refuses rather than warns.

The purpose of an evaluation system is not to produce convincing numbers. It is to reject numbers that cannot survive the conditions under which the system must operate.

Appendix: manifest

Scope

One platform, one domain, one author, and eleven documented failure cases of several related classes; there is no claim that these are all the failures the platform has had, only all the ones it found and recorded. The cases were built by AI coding agents under the platform's standards and found by the same agents once the questions and then the gates existed, which the article on that program covers. The mechanism is general: ordinary batch scoring of any sequential decision system holds the whole sequence, and the classes in section 4 appear in recommendation, ranking and forecasting pipelines in the same forms. The harness and the policy are offered as the smallest set of checks that caught every case here, not as a proof that nothing remains.