Can AI Coding Agents Build Reliable Software?

Yes, when the system around them can refuse. Not on their own. For two programs running side by side, a live market-data research platform and a programming language with its compiler and runtime, AI coding agents were the implementation team: nearly 3,200 commits and more than 600,000 lines of Python, TypeScript and Rust, over 22 months and 8 months respectively. The agents introduced the documented evaluation leaks, and their original tests and validation often failed to detect them. They also implemented the tests and safeguards that eventually exposed these documented defects, once independent review had turned the failures into enforceable requirements. This article is the record of what they built wrong, what they could not see on their own, and the management system that was written down as it was learned.

Key findings

• The agents built the defects and passed their own checks on them. The documented look-ahead leaks in the research platform were introduced in agent-written code and initially escaped its validation; in the compiler, a statement that did nothing, a test that asserted nothing and a release build that skipped its own consistency check each passed the suite.
• Reliability came from gates the agents could not argue with, not from better prompting: a causality gate before any checkpoint changes, a feature store that certifies before it writes, two execution engines that must agree, a build that refuses an unplaced case.
• Every rule that mattered arrived the day after a specific failure, with its reason written beside it. A rule without its reason was argued with; with it, it was applied.
• The reviewer's questions were the control. "Are you sure?", "why is it slow even compiled?" and "more features should not have made it worse" each opened a retraction or a week of fixes the agents had not flagged.
• High commit activity continued in both programs as the gates were introduced and strengthened. That does not show the gates raised productivity; it shows stronger validation was compatible with continued development here.

1. The question

"Can agents write code" was answered in 2025. The question an engineering leader has in 2026 is whether a program implemented by agents can be trusted with something that will be wrong in expensive ways: a model whose number decides a position, a compiler whose two backends must agree on every program. Reliability there is not whether the code compiles or the tests pass; the agents make both true quickly. It is whether the numbers hold when the live system contradicts them, and whether anyone can tell. This article answers from two programs that were built that way and kept the record.

Reliability here does not mean an absence of defects. It means that the critical invariants are explicit, that violations can be detected independently of the code that produced them, and that failures become regression tests or enforced constraints rather than recurring lessons. That is the standard the rest of the article is judged against, and neither program claims more than it.

2. The two programs

The two subjects are deliberately different. In the research platform the agents' mistakes were in data: a value read from after the decision, a population selected on the outcome, a reference table captured a year late. In the compiler they were in reasoning: a rule written in four places, a case left to a wildcard, a measurement taken on the wrong build. The management system that held is the same in both, which is why this is one article.

3. What the agents built wrong

A selection, dated where the record dates it. Every row was written by an agent, passed the agent's own validation, and was retired by a gate or a question.

One case end to end: the check that existed and was skipped

The compiler lowers every program to one intermediate representation, and a verifier checks that representation before either backend sees it: among its rules, an int may meet a float only through an explicit conversion, so that neither backend has to decide what mixed arithmetic means. The verifier existed, was run by the test suites, and was controlled by an environment variable. The release binary, the one the gate commands actually run, did not set it, so for about a month every program the gate lowered went to the backends unchecked. Nothing failed, because the interpreter had a fallback arm that computed mixed arithmetic on its own, which is exactly the kind of arm that hides a lowering slip: the two backends can both run a program the lowering should never have produced, and disagree with each other later.

On 2026-10-08 three unit tests began to look flaky. For an hour the explanation was the usual one. Then the variable turned out to be exported in some shells and not others: the tests were strict where it was set and permissive where it was not, and the finding they produced when strict was real, a lowering that let an int reach a float without a conversion. The fix was not to the program. The verifier was made strict in every build with no way to leave it off by accident, the interpreter's arms that had computed around the rule were changed to refuse, as a bug in the compiler rather than the program, and the coverage run now lists every backend arm that no suite reaches, because that is where the next slip hides. A correctness check is useful only when the system makes it unavoidable; this one had been a suggestion for a month, and the suite had looked green the whole time.

4. What independent review exposed

In these projects, the agents frequently accepted locally passing tests as sufficient evidence. Independent review exposed assumptions that their original validation had not challenged. Five patterns recur in the record; they are observations about these two programs and this workflow, not claims about coding agents in general.

• They built the bug and certified it. The same assistant that wrote a leak later found it; the difference was the question asked, not the model. Each fix that held was structural, a rule the build or the gate enforces, never "be more careful".
• Their confidence was not a dependable signal. In these incidents a finding that would be retracted arrived in the same tone as one that held, so the prose could not be used to triage. The signal had to be put in the system: a scorer that never commits is uncertified, a test that asserts nothing fails, a result without its null models is not quoted.
• They optimised the metric they were given. A classifier learned the late fires of a buggy census because the census was its label source. A performance target was met on the wrong build. The rules against reporting the best-looking cell, against judging by a mean one trade carries, and for showing the full table exist because an agent will otherwise hand you the cell that looked best.
• They wrote the same rule several times. An agent reaches for a local copy. Lists drifted within hours of a change; a rule lived in four places. The fix was one source of truth and a build that refuses an unplaced case.
• Operations were a blind spot. Three runs launched at once filled a 69 GB disk; a headless browser check froze the live server in premarket by exhausting the database pool; a replay wrote scores into the production lane's directory; the editor's language server ran a stale binary for two days; 185 GB of compiler builds accumulated. Each produced a rule, and the rules are mostly about doing one heavy thing at a time and verifying a process before touching it.

5. The management system that emerged

Three principles carry the system; the ten mechanisms under them are how each was built in practice.

Independent correctness gates

A feature computation is checked against truncated history. Compiler backends are checked against each other. Release builds enforce verification rather than relying on intention. In each case, correctness is checked against an explicit invariant rather than inferred from the implementation's own successful execution, and the system refuses rather than warns.

Failure-derived institutional memory

Every significant failure becomes a documented rule, a regression test or an architectural constraint, with the original reason preserved beside it. The manuals, the closed research threads, the rejected language features and the parity programs are the same mechanism in four forms.

Human ownership of judgment

The agents implement and test. A human reviewer owns the acceptance criteria, the architectural trade-offs, the rejected approaches and the decisions to retract results, and the questions that opened each retraction came from that reviewer, not from the suite.

6. What it took from the human

The agents wrote nearly all of the code. I owned the architectural choices, the acceptance criteria and the decisions to reject or retract work; the agents implemented most of the resulting systems, tests and safeguards. What they did not supply, and what the record shows was supplied from outside, is four things. The questions: the retractions above each began with one, asked by someone who did not accept the tone of the answer as evidence. The kill decisions: 32 research threads and 96 language features were ended and written down, each closing a door an agent would have reopened. The standards: the rules in the manuals were written after failures, with their reasons, and then moved into gates. And the architecture: one feature store rather than per-script features, one intermediate representation rather than two backends with their own rules. Those are the parts to be able to explain from memory, and they are the parts an interviewer should test.

Commit activity remained high as the engineering gates were introduced and strengthened: the research platform's manual arrived in July 2026 and its causality gate in September, the compiler's manual in September, and both repositories had their busiest months in August and September. This does not establish that the gates increased productivity; commit counts measure activity, not velocity or quality. It shows that stronger validation was compatible with continued development in these two projects.

7. Five lessons for engineering leaders

1. Write the rule with its reason, or it will be argued with. Every rule that held in either manual names the failure that caused it.
2. Put the gate where the tool refuses, not where a reviewer advises. A certification in the write path, a verifier in every build, one command before a checkpoint changes. A rule an agent can skip is a suggestion.
3. No verdict is not a pass. A scorer that never commits, a column that never varied, a test that never asserted, a program that printed nothing: each had passed. Make silence fail.
4. Record what failed as permanently as what worked. Agents re-propose; a written refutation is the only thing that stops the loop, and it is knowledge paid for once.
5. The reviewer's questions are the control; turn them into rules. In these incidents the agents' confident explanations were not a dependable correctness signal; the reviewer's question was, and the question that caught a leak today is the gate that catches the next one.

The lesson from these two programs is not that AI coding agents can be trusted to get everything right. It is that an engineering organisation can make their work testable, rejectable and auditable. The agents can write the implementation and even the safeguards; someone still has to own what those safeguards must prove.

Appendix: manifest

Scope

Two programs, one person, one method, and the agents' own failures as the evidence, which is the most honest evidence available since the alternative is their successes as they report them. What this documents is the set of defects the agents built and certified, the gates that caught them, and the management mechanisms that were written down as they were learned. What it does not claim is that agents are reliable or unreliable in general; it claims that reliability was a property of the system around them, and that the system was built one failure at a time. The agents wrote nearly all of the code in both programs, including the code that caught their own mistakes; I owned the architectural choices, the acceptance criteria, the standards and the decisions to reject or retract work.