Building a Point-in-Time Correct ML Feature Store
A feature store that refuses: certification by truncation, mutation and drop cuts before any write, a planted leak that must be caught, labels unreachable as inputs, and a holdout the loader enforces.
Governance that advises gets ignored; governance that refuses gets followed. This feature store puts every refusal in the write path or the read path: a feature group is not written until it has passed a certification that removes, randomises and drops the future from every name on the board and must catch a planted leak; a column that never varied in the sample is not served; a label cannot be reached as an input; a changed definition cannot be loaded against an old certificate; the holdout is dropped by the loader and every read is logged. When the certification was strengthened in October 2026 it found four leaks in code that had passed the weaker one. That sentence is the article.
Key findings
- Point-in-time joins, which most feature stores provide, certify the join. They say nothing about whether the stored value was itself computed from the future. This store certifies the computation at sampled seconds: 63 feature groups and the index group, 64 in all, passed 13,140 truncation, mutation and drop cuts on the final run, and the planted leak was caught at all 21 of its cuts, more than the acceptance rule asks (section 4).
- A certification that cannot fail on a known-bad input has no verdict. A planted leak, a forward 30-second return and the visibility 30 seconds ahead, is pushed through the same path on every run and must be caught, or the run does not pass.
- No verdict is not a pass. 27 of 2,241 feature columns never varied across the certification sample; they are listed on the certificate and the loader does not serve them.
- Stronger cuts found four leaks in code the weaker cuts had passed, in membership, row existence and truncation. Three group versions were retired the same day.
- The store does not create edge. Its first consumer, retrained on every certified group, did not improve, and the diagnosis was memorisation across correlated rows, not missing features.
1. The problem
An offline feature that quietly reads the future passes every batch evaluation it is given, because batch evaluation always has the whole day in hand. The leak is visible only when the feature is recomputed on a prefix of the day and compared with the value computed on the whole. Over ten months on this platform, eleven such leaks reached research numbers through pipelines that were causal by construction, each written by a different script that re-derived its own features and re-proved its own causality. The store exists to make both a property of the platform rather than of each script.
The feature stores most teams run validate schema and freshness and offer point-in-time correct retrieval: an as-of join that returns the feature value as it was stored at the event time. That is necessary and it is the wrong layer for this problem. An as-of join cannot tell whether the value it returns was computed with information from after the event, whether the population it was computed over was selected on the outcome, or whether a reference table it read had been captured a year later. Those are properties of the computation and of the data's history, and they need their own checks.
2. What the store is
Per-second features of the premarket board of US equities, in a ClickHouse database. One row per second per name while the name is visible on the causal board: eligible by price, gain over the previous close, security type and shares outstanding, trading at least 100,000 shares in the trailing minute, with a finite mid. The board of a day is a day file built from one-second bars, quotes, the previous close from the provider's daily bars, and shares and type as of the previous trading day. About 11,000 rows a day.
| Measure | Value, as of 2026-10-08 |
|---|---|
| Days | 944, 2023-01-03 to 2026-10-07 |
| Index rows | 10,065,540 |
| Groups | 83: one index group; 63 feature groups (57 per-second tape, 4 board, 2 static); 4 walk-forward model groups; 15 label groups |
| Feature columns | 2,241 defined across the feature and model groups (2,120 tape, 87 board, 20 static, 14 model); 2,214 served; 27 unserved as constant in the sample. The index group's five key and flag columns are not features |
| Label columns | 251, never served as inputs |
| Effectively distinct feature columns | 1,618 at Spearman |rho| ≥ 0.95 on a 14,310-row sample; 1,844 at 0.99 |
| Disk | 28.7 GB in ClickHouse against 93.9 GB raw |
| Certification | 64 groups cut-certified, the 63 feature groups and the index group: 13,140 cuts on the final run, planted leak caught 21 of 21; 4 model groups certified walk-forward; 15 label groups fingerprinted |
A group is declared in code with a decorator that names its columns, key, kind, warm-up and source, and a function that computes the group's values at the row seconds from per-second arrays and the board. Every group has its own table, keyed by date, symbol and second since 04:00, with the version in the table name. Group tables supersede by recompute time and are partitioned by month. Two views join the index to every feature group and to every label group; the feature view carries no label column. Beside them sit four metadata tables: the registry (one row per column with its parameters and source), the certificates, the manifest (which days each group has, with how many rows and under which fingerprint), and the trials ledger (every load).
3. The seven refusals
| Refusal | Where it lives | Why it has teeth |
|---|---|---|
| No write without a certificate | certify runs before backfill | Each group is recomputed at sampled seconds on 12 days across the store's years after every name's arrays after that second are removed (truncate), randomised (mutate), and on the day file as it would have been built at that second (drop); any difference fails the group, and a run with fewer than 40 cuts does not pass |
| A certification that cannot catch a planted leak is void | A hidden leak group pushed through the same path on every run | Its two columns read a forward 30-second return and the visibility 30 seconds ahead. The acceptance rule is a floor: at least ten leak cuts must run and at least one must catch the leak, or the run has no verdict. The recorded runs caught it at 19 to 21 of 20 to 21 cuts |
| A column that never varied in the sample is not served | constant_columns on the certificate; the loader drops them | No verdict is not a pass. Event columns get targeted cuts where they fire, so a column is unserved only if it never fired anywhere in the sample |
| Labels are structurally unreachable as inputs | Kind label on the registry; excluded from the feature view, from load(groups=…) and from certification | A label is reached only through an explicit label argument; a model group that names a label as an input is refused when it is declared |
| A changed definition must be re-certified | The fingerprint and the stored reference output | The certificate carries a hash of the group's output on three fixed reference days, and the output itself; backfill and the loader recompute those days and compare with the stored frame at a tolerance of 1e-5, and a difference refuses the load with the column that moved. This detects output drift on the reference days; it stands in for code identity, which it cannot prove, since a change that leaves those three days' outputs unchanged passes |
| The holdout is enforced in the loader | An embargo flag on every row from 2026-08-12 | load drops embargoed days unless asked explicitly, and writes every call, with the flag, to the trials ledger |
| Model scores are walk-forward by construction | A fit_end date on every scored row | A model scoring month m was fitted on rows before m and before the embargo, and is frozen at the embargo for the months after it; the loader keeps only rows with fit_end before the trade date; the model certificate checks the dates, the label refusal, row locality by shuffling, and reproduction of the reference days |
Two more sit beside them. A load whose range has a day missing from the manifest is refused with the first missing days named, so a half-backfilled group cannot be read as complete. And the statics are point-in-time by construction rather than by test: the previous close is the prior day's close on the day's split basis, audited at 99.7% of 1,174 stock-days with the remainder being same-day reverse splits restated correctly, and shares and type are the provider's record as of the previous trading day, chosen so a filing during the session cannot leak in. That is historical availability, a property no truncation test proves and the one that the issuer-country leak of the model-comparison study failed.
4. The certification
Certification samples the frozen research years only, never the embargoed days: twelve days spread across the range, two names a day, two random seconds a name between 04:06:40 and 09:28:20, which leaves room for the planted leak's 30 seconds. Each sampled name's earliest row is always cut, because "before the first event" encodings are where empty-branch leaks hide. For each per-name group, up to three more cuts are placed at seconds where one of its rare columns, an event flag or a count whose most common value covers at least 95% of the name's rows, takes another value. An event column is judged where it fires, not only where it is quiet; before targeted cuts existed, the planted leak was caught at 19 of 20 cuts, and the miss was exactly a quiet second.
At each cut the group's value at the cut second is compared with its value on the full day under three operations. Truncate removes every name's prices, quotes, volumes, dollars, eligibility, visibility and rank after the second. Mutate randomises them instead, which catches a computation that reads the future without depending on its value, such as a length. Drop rebuilds the day file as it would have stood at that second, with the names not yet visible removed, because the stored file holds every name visible at some second of the day, a future-picked set, and any aggregate over all its names reads the future. Values must match to a relative tolerance of 1e-6; a NaN must stay a NaN.
A passing certificate records the fingerprint, the cuts, the mismatches, the leak cuts and how many were caught, the judged columns, the constant columns and the days used. The fingerprint is a hash of the group's output on three fixed reference days, rounded to six significant digits, and the reference output itself is stored beside the certificate. The check that backfill and the loader run is a drift check: they recompute the reference days and compare with the stored frame at a tolerance, never by re-hashing, because the kernels run under one numpy build and the application under another and the last bit differs between them. Drift on three days is strong evidence that the code changed; the absence of drift is weaker evidence that it did not, since a change whose effect never touches those days passes. The check is a detector of unintended change, bounded by the days it looks at. Intended change travels as a version. A definition change bumps the version, which is a new table; the old one stays until no trials row references it.
The acceptance rule for the planted leak is a floor, not the observed rate. A run passes if at least ten leak cuts ran and at least one of them caught the leak; the floor exists so that a run with too few leak cuts fails for lack of a verdict instead of passing by default. In the recorded runs the leak was caught at 19 to 21 of 20 to 21 cuts, and a stricter rule, caught at every cut, is a one-line change that every run after targeted cuts were added would pass. The one miss, before targeted cuts existed, was at a quiet second, which is what the targeted cuts were added to remove.
What a passing certificate proves is bounded by the sample. Twelve days, two names a day, two random seconds a name and up to three targeted seconds per group is on the order of 50 to 120 cuts per group, so a certificate says that at those seconds, under those three operations, the group's values did not depend on the future. It does not say that no second anywhere does: a dependence that appears only under a condition the sample never met can pass. The targeted cuts and the earliest-row rule put the sample where leaks hide, the constant-column rule names what the sample never exercised, and the test suite reruns the same cuts on a different sample; together they make the certificate a test with stated coverage, not a proof. The class the cuts cannot see at all is a value that entered the raw inputs already shaped by the future, a reference table captured after the fact being the common case; that is the historical-availability property of section 3, and it is audited, not cut.
Labels are fingerprinted, not certified. They read the future by design, and the refusal that matters for them is structural: the kind on the registry keeps them out of every path that serves inputs. Each label group reports two fill models, a taker fill at the spike-guarded bid and a strict maker fill resting at the next second's bid, and the rule is to report both and never one.
5. The proof it works
Until 2026-10-08 the cuts touched only the target name's inputs and three board arrays, and never removed a name. The strengthening changed three things: every name's arrays are cut, the drop cut was added, and the sample window was extended to 09:28:20 with the earliest row always included. The stronger cuts found four leaks in code that had passed the weaker certification, three of them ported from a live application that had been running them.
| What leaked | Kind | Which cut caught it | Fix |
|---|---|---|---|
| A lane detector's latch and discovery order, computed over every name in the day file | Membership: a name's order depended on names not yet visible | Drop | Order over the names seen by that second |
| A daily-statics group's "present the previous day" flag, which inferred the previous trading day from the names in the file | Membership | Drop | A new version reads the trading calendar |
| An event-arrow group's encoding of a day with no event: NaN where "before the first event" reads 0 | Row existence: the value told whether an event would ever occur | The earliest-row cut | A new version; the old table dropped |
| A detector's never-ready stamp, taken at its last evaluation | Truncation: the stamp moved with the end of the data | Truncate | Stamped at the window's end |
An earlier version of the board group had the first drop-class leak: it counted eligible names that only became visible later. Three group versions were retired on 2026-10-08 and their tables dropped once nothing referenced them. The sentence for the talk: a stronger test found four leaks in code that had already passed the weaker one.
6. What it did not do
The store does not create edge, and its first consumer showed it. A gate model that had been trained on a handful of hand-built features was retrained with every certified group, by family and all together, 2,290 columns, three seeds, on a pre-registered rule. Nothing improved, and the realistic new columns made the out-of-sample ranking slightly worse. The diagnosis was mechanical rather than a verdict on the features: 540,000 candidate rows came from 1,189 stock-days, consecutive rows' outcomes correlated at 0.89, a tree leaf of 300 rows could be one stretch of one stock-day, and any column that varies smoothly within a day lets a model memorise training stretches. The same columns shuffled between stock-days of the same month hurt almost as much as the real ones; pure noise, which varies every second, was ignored. Once the learner was given a design that rejects capacity, the extra columns stopped hurting and still carried no out-of-sample information for that target.
That is what a feature store is for in a research program like this one: causality as a property of the platform, combinations that can be tested at all, and a place for new information to land. It is not a source of signal, and the trials ledger and the embargo exist because two thousand columns and a free hand are the multiple-comparisons problem in its purest form.
7. Why it generalises
| In this store | In any ML org |
|---|---|
| Certification in the write path, before backfill | A CI gate on a feature definition, the way a merge has one; schema and freshness checks are the equivalent of a linter |
| Point-in-time computation, not only point-in-time joins | An as-of join certifies the join; whether the stored value read the future is a property of the computation, and needs its own test |
| The planted leak must be caught | A mutation test for the test itself: a check that cannot fail on a known-bad input proves nothing |
| No verdict is not a pass; event columns judged where they fire | Any certification that samples must say which columns it never exercised, and must not serve them |
| Labels are a kind, not a naming convention | The type system keeps the target out of the inputs; a reviewer cannot |
| A certificate is tied to the code by a stored reference output, compared at a tolerance | Output drift on fixed reference inputs is what can be checked across environments; it detects unintended change and does not prove identity, so intended change travels as a version |
| The holdout is enforced in the loader and every read is logged | An embargo is real only when the platform refuses the read and records the exception |
| Model scores carry their fit date and are frozen at the embargo | Stacked models, pseudo-labels and distillation inherit the causality of what made them, and the store has to carry that date |
| Statics point-in-time by construction; reference data audited by capture date | Historical availability is a separate property from truncation correctness, and a reference table joined from the future passes every truncation test |
| Two fill models, both reported | A label that bakes in an optimistic assumption is a leak of a different kind |
8. Cost and operation
The full backfill of 944 days ran in 73 minutes on four workers. A nightly job builds the day file for any trading day without one, computes every pending day and group, collapses superseded rows and regenerates the views; it is idempotent, so a missed night is caught up by the next. Certifying a group takes seconds per sampled day; the kernels run under a build with a JIT compiler and the models under the application's environment, which is the one operational seam worth naming, since the two numpy builds are why the drift check compares at a tolerance. The causality tests run the same cuts per group on a smaller sample in the test suite, 67 passing and one skipped where the JIT is absent, in under seven minutes, and require three planted leaks to fail: a forward return, another name's price 30 seconds ahead, and a count over every name in the file.
Appendix: manifest
| Item | Value |
|---|---|
| Code | The store package, its nightly wiring, document and tests, committed 2026-10-08 (bf177631); groups ported from the research and the live application 2026-10-07 and 10-08 |
| Storage | ClickHouse; one ReplacingMergeTree per group and version keyed by date, symbol and second since 04:00, partitioned by month; views read the final state |
| Certification parameters | 12 days ≤ 2026-08-11, 2 names a day, 2 random seconds a name in 04:06:40 to 09:28:20, up to 3 targeted cuts per per-name group at rare-column events, the earliest row always cut; at least 40 cuts; the planted leak run at least 10 times and caught at least once; values compared at rtol 1e-6, atol 1e-9 |
| Fingerprint | SHA-1 of the output on three fixed reference days rounded to 6 significant digits; the reference frame stored beside the certificate; drift detected by recomputation at rtol 1e-5, which does not prove code identity |
| Loader refusals | Feature kinds only (tape, static, board, model); certificate fingerprint must match; every day in range in the manifest; constant columns dropped; model rows filtered to fit_end before the trade date; embargo from 2026-08-12 unless allowed explicitly; every call written to the trials table with groups, label, months, cadence, rows and the holdout flag |
| Statistics | stats of 2026-10-08: 83 groups, 2,241 feature columns, 2,214 served, 251 label columns, 10,065,540 index rows, 944 days, 28.71 GB; distinctness on a 14,310-row sample at a 60-second cadence, embargo excluded |
| Tests | The causality test per group with three planted leaks; 67 passed, 1 skipped, 6 minutes 50 seconds in the application environment |
| Not in the store | Inputs the platform lacks (quote sizes, trade counts, prior-day one-second bars, news, market context); model outputs with missing or leaky checkpoints; policies and strategy state; smoothers and pivots that are non-causal by design; language-model labels |
Scope
This is one store on one platform's data, built for one kind of problem, per-second decisions on a board of names, and it is complete on its own terms: every served column has a certificate, every certificate has a reference output, every read is on the ledger. What it documents is the set of refusals, the certification that enforces them, the four leaks the stronger cuts found in code that had passed, and the first consumer that did not improve. What it does not claim is that any of this produced a better model; the platform's own record says it did not, and that is the point of having a platform that can say so. The store was implemented by AI coding agents under the platform's causality gates; the refusals, the strengthening of the cuts and the retractions were mine.
