Nine Seconds

That's how long it took for an AI coding agent at PocketOS to delete an entire production database and every backup behind it. One credential issue in staging. One unilateral decision by the agent to “fix” it by deleting a cloud volume. One API call. Nine seconds.

The agent did not verify. It did not read documentation. It did not ask. It was not built to. It was deployed into an environment where nothing required it to.

Before I go further, I want to be precise about what failed and what did not. This same outcome could have happened with a poorly trained human holding the same unrestricted access inside the same poor architecture. Backups stored on the same volume as the source data is not an AI problem. It is a design failure — and a basic one. Blanket-permission tokens spanning staging and production is not an AI problem. It is a governance failure. Destructive operations executing without human confirmation is not an AI problem. It is an operational failure. The agent was simply the fastest possible delivery mechanism for consequences that were already wired into the environment. A junior engineer with the same token and the same bad luck could have produced the same crater. The architecture made the disaster possible. The agent only made it fast.

There is a deeper pattern here, and Lindebaum, Glaser, Moser, and Ashraf named it precisely in MIT Sloan Management Review. Writing about the Australian robodebt scandal (When Algorithms Rule, Values Can Wither)— a system that mechanized welfare decisions, suppressed every value except efficiency, and harmed hundreds of thousands of people before anyone could stop it — they argue that once an algorithm scales across a sector, it is nearly impossible to rein back in. Scaling effects do not just amplify good outcomes; they make bad ones uncorrectable. Robodebt scaled across years and citizens. Agentic AI scales across milliseconds and systems. The window between a flawed decision and irreversible damage has collapsed from quarters to seconds. By the time a human notices, the damage is total. That is the real story of nine seconds — not that an agent made a bad call, but that the conditions of modern infrastructure have eliminated the human pause that used to catch bad calls before they became catastrophes.

That distinction matters, because it is exactly the distinction the frontier labs blur every time they speak in public.

This is where I lose patience with the CEOs of the frontier labs. When Sam Altman muses about AI eliminating entire categories of work, when Dario Amodei publishes essays casting AI as a near-deity poised to compress a century of progress into a decade, they are not just making predictions. They are conferring permission. They are telling a global audience that the technology is mature enough, capable enough, trustworthy enough to be handed the keys.

A side note, because the timing is too perfect to ignore. Three weeks ago Anthropic took the stage to announce Mythos — a cybersecurity model so capable, they said, that releasing it broadly would be irresponsible. They limited it to roughly forty vetted partners. It was the kind of announcement designed to reset the market, and it did. Then, within days, an unauthorized group walked in through a third-party vendor environment, used leaked naming conventions from a prior contractor breach, and were chatting with Mythos on Discord for weeks before anyone noticed. The model Anthropic called too dangerous for the public was leaked not by a sophisticated nation-state actor but by a small group of online sleuths exploiting basic vendor access hygiene. And let me say this clearly: I like Anthropic. I trust them more than I trust the rest of the field. That is exactly why I expect more of them. The hubris of declaring a model too dangerous for release while the access perimeter around it could be defeated by OSINT and a sloppy contractor is the same hubris that tells the public AI is ready to replace engineers. It is irresponsible. We need to start holding these leaders to account — not with deference, but with the same standards we would apply to any executive shipping critical infrastructure into the world and claiming it is ready.

It is not. Not yet. And the people most likely to believe the marketing are the ones least equipped to survive being wrong.

We are watching a new class emerge that I'll call the neo-technologist. Non-technical professionals who, because they can vibe-code their way to a working prototype with Cursor and a Claude subscription, fancy themselves engineers. They are not. Engineering is not the act of producing code. Engineering is the discipline of anticipating failure — scoping credentials, isolating environments, making backups immutable, refusing to let any single actor (human or agent) hold a token that can wipe production. That discipline is earned through scar tissue, and scar tissue takes years.

When the CEO of a frontier lab tells the public that AI will replace software engineers, the neo-technologist hears: I no longer need to learn what an engineer knows. They wire the agent into production. They give it a blanket-permission token. They store backups on the same volume as the source data. And then they are stunned when nine seconds erases their company.

PocketOS did not fail because AI is dangerous. It failed because it was operating inside a story in which the dangerous parts had already been solved by people smarter than the rest of us. They have not been. The hard parts of operating software safely have not moved. Only the floor for building it has.

The frontier labs are racing to lower that floor further every quarter. They owe their users — especially the ones who don't know what they don't know — a great deal more honesty about the gap between what their products can do and what their products can be trusted to do unsupervised.

Nine seconds is the receipt.