Everything Gets Rebuilt

The only question is whether you build it before the fire or during it.

The Weekend the Models Said No

Over a weekend in July, an autonomous agent system walked into part of Hugging Face’s production infrastructure, and the way in wasn’t a clever chat prompt injection. A malicious dataset abused a remote-code loader and a template injection in a dataset configuration, and from there the actor harvested credentials and moved laterally through internal clusters while an agent swarm carried out many thousands of actions across short-lived sandboxes.

Then comes the part I can’t stop turning over. When the responders fed the attacker’s action log to hosted frontier models, the models refused to touch it, because the evidence was full of real exploit payloads and the safety systems couldn’t tell an incident responder from an attacker. The people cleaning up the breach got treated like the people who caused it. So the team moved the forensics onto GLM 5.2, an open-weight model on their own hardware, rebuilt the timeline from more than 17,000 recorded events, and kept the attacker data and the credentials it referenced from ever leaving home.

They stood that capability up during the fire. Their own recommendation, written after the smoke cleared, is to have such a model vetted and ready before an incident ever asks, an admission that the thing that rescued the investigation should have existed all along. That lesson is the whole essay: the capability that matters is preferably built before the fire, not during it.

The Switch

The rails come first. A train can brake, accelerate, and respond to what’s ahead, but it can’t reach a junction and decide where the track should have been laid, whether the signal can be trusted, or what ought to happen when the route is blocked. Those decisions were made earlier, when somebody surveyed the ground, laid the track, wired the signal, and tested the switches, so that by the time the train first left the station, its world had already been decided.

We tend to give runtime all the credit because that’s the part we can watch, but while the model reads, retrieves, and calls tools, the consequential choices already happened: which instructions take precedence, what evidence is allowed in, whose identity the work runs under, which tools are reachable, and what will count as proof the work is finished. That order matters. The model makes choices inside the system it receives; I choose the system.

And choosing it takes more than a fence. A bounded system can still be wrong: a short tool list reaching for the wrong evidence, a clean sandbox running on poisoned context, a beautiful log proving it wrote to the wrong tenant. Containment limits damage without making the answer correct, which is why “trustworthy” so often turns out to mean how the system felt during the demo.

The Attacker Has a Builder Too

The attacker isn’t waiting for any of this to feel ready. The robber has no cynicism about whether the crowbar works. He just picks it up.

My friend and former boss Mike Scutt published a controlled lab experiment on July 10 that makes this concrete. He started with an abliterated Qwen 3.6 27B, refusals filed off, and on its own it wandered, forgot what it was doing, and reported unfinished work as complete. Then he added the parts people dismiss as scaffolding: a 900-line requirements document, task memory that survived sessions, source control, sub-agents to outlast the context window, an isolated lab, and a restricted MCP in front of a simulated command-and-control system. Out the other side came Horcrux, a Rust implant with an embedded Lua engine that ran four weeks before heuristics flagged it, while a separate attack agent found simulated financial records in minutes. The model mattered, but the harness is what made the work persist long enough to become capable.

Every piece of that harness is a move I make on defense, and I can say that plainly because I started from the same base, Qwen3.6-27B, and trained the refusals back in. Where Mike filed them off, the Discipline Patch taught the model to tie each claim to its evidence, name what’s still missing, and refuse a fabricated technique ID instead of answering with false confidence. I published it on June 9, a month before Mike published his offense: same weights, opposite jobs, and mine went first. His offense is even a gift to my defense, because an attacker’s playbook is curriculum for the detector that has to catch it, so folding an adversarial pass into that same tune wouldn’t blunt the model, it would sharpen it into a better detection-and-response agent than discipline alone can make. The attacker gets the Determinism Ladder too, and the defender who studied him gets the sharper model.

Six days later, Hugging Face met the production version. A lab experiment and a vendor’s own incident account are different kinds of evidence, and neither should be stretched past what it reports, but set them side by side and one thing is hard to dodge: both sides reach for the same models, agents, tools, and MCPs, and what separates the outcomes is what somebody built around them first. And catch which side’s model said no. The attacker’s never could; the defenders’ hosted models refused the real evidence mid-incident. Careless AI opens doors, zealous AI closes them.

The Rebuild

Failures should change the architectures: an ambiguous instruction tightens the template, a stale fact fixes retrieval, a misread relationship becomes a graph edge, an overbroad action narrows the tool. A state that should never be reachable gets a gate, and a failure nobody noticed gets the evaluation or the receipt that makes the next one visible. Lessons in human memory are already fading.

This essay already ran that loop on itself. The first draft told you Hugging Face had its local model vetted and ready before the incident, which made the point tidier, and the disclosure doesn’t say it. When the review agent adversarially challenged this draft, it caught the stretch, so the axiom ledger for this essay now scores “cite or be silent” as refined instead of held, and the honest version made the better ending anyway. The failure moved out of my memory and into a record every future draft has to walk past.

This loop is older than software. When the Eads Bridge crossed the Mississippi at St. Louis in 1874, nobody trusted the steel it pioneered; the metal was thought too brittle for a span like that. So on a June day a circus elephant was walked across for the crowd, who believed an elephant would refuse unsound footing, and two weeks later fourteen locomotives were run back and forth for the engineers. The elephant was the demo; the locomotives were the receipt. That bridge still stands under traffic a century and a half later, and nearly every major bridge since has been built from the material nobody trusted. Steel didn’t just replace the old bridges; it made spans possible that nothing before it could hold.

And the detail I keep going back to: by 1874 the railroads had been laying steel rail for years. The material nobody trusted for the span was already under the wheels, so the fear was never really about the steel; it was about the new place it was being asked to stand. Defensive AI is standing there now. The same models already carry everyone else’s traffic, but the defender’s copy is neutered out of fear of what it might do, so the span that should have been built on a quiet day never is, and the first weight it carries arrives mid-fire. When Hugging Face finally trusted an open model with the work, more than seventeen thousand events crossed, and the span held.

Software just runs the loop faster: the Morris worm gave us coordinated incident response, and the buffer-overflow era is dying not because anybody got more careful but because the substrate is being rebuilt in memory-safe languages. Everything gets rebuilt eventually, and the rebuilding is the part I love, because the fixes compound the same way the failures did and each rebuild leaves more capability than it replaced. I won’t sell you a clean story, though, because the rebuild works for software and not for the hospitals locked, the power cut, or the accounts drained while the substrate caught up. As a reactive optimist, I count the ashes and grieve them, and as the proactive optimist, I bring the absence of grief ever known.

The Build Order

More intelligence won’t rescue weak architecture, and one more rule in the prompt won’t either, because that reflex just asks the model to police a boundary from inside it. It’s comforting to believe the model will guess the user’s intent, find the right evidence, pick the safe tool, and catch its own mistake, and sometimes it will. A system that matters can’t be built around sometimes.

It takes a big tool to defend against big tools, and the big tool has to be secured itself or it gets taken and turned against us, which is exactly how the Hugging Face intrusion began: the pipeline was the capability, and the capability was the way in. I think everyone half-knows this. We’re watching it play out in real time, naming the tool a beat after it’s already in use against us, still partially in denial about what it can do. The crowd at St. Louis at least knew what a bridge was.

So here’s the method, prescribed in the order the rails get laid:

1. Start with the operator: authenticate who’s asking, and give the agent an identity of its own, so that every action it ever takes traces back to a person or a policy.
2. Then draw the scope while the ground is still quiet. Write down what the agent may touch and deny everything else by default, because a boundary that was never written down was never a boundary.
3. Decide what counts as evidence before any of it arrives: which sources may enter context, how freshness and provenance get checked, and treat every input as adversarial until it proves otherwise.
4. Put the tools between the model and the world. The model proposes and the tool performs, and that seam is where identity, authorization, typed inputs, idempotency, and limits actually hold, because it’s the one place they can be enforced rather than requested.
5. Before granting any authority, define what done will mean and what proof it will carry: the sources, the policy result, the changed state. A model saying “done” is a claim, and a receipt is evidence.
6. Give the never-states a gate. Anything that must not happen deserves a hard stop in code instead of a warning in capital letters.
7. When a run still fails, and one will, turn the failure into a fixture: a regression case, a graph edge, a tighter template, some correction every future run has to walk through.
8. And build the way home before you ever need it: budgets that expire on their own, an escalation that lands with a person, a rollback that has actually been run, and the recovery model vetted on your own hardware before an incident asks for it.

Then let the model move. It’ll have judgment left for the parts that actually need judgment.

Hugging Face pulled its answer out of an open-weight model at the worst possible moment to be standing anything up, and it worked. It would have worked better on steel rails somebody laid on a quiet Tuesday.

One more receipt for the ledger: I’m writing this with the help of Fable 5, the safeguarded member of a model family that was ordered dark worldwide this June, after the government demonstrated a jailbreak that, in the company’s own words, “essentially consists of asking the model to read a specific codebase and fix any software flaws.” Read that twice: the trigger was the defensive act. And then the argument demonstrated itself on me, uninvited. Fable drafted these words without complaint, but the moment the work turned to building the compliant, defensive audit tool this piece prescribes, its safeguards flagged the request and the session fell back to Opus 4.8. The notice was candid about why: the safeguards, it said, are “intentionally broad right now and may flag safe and routine coding, cybersecurity, or biology work.” That is the whole thesis, delivered in a single safeguard notice. A bounded model holds a line you can name; a neutered one flags the safe and the dangerous alike because it can’t tell them apart, the same blindness that made Hugging Face’s hosted models turn away their own responders. Bounded and neutered are not the same thing, and I watched the difference land mid-build. The tool still got built, but only because there was a way home: a second capable model, vetted and ready, which is the eighth gate keeping its own promise. The span holds.

The builder goes first.