A Switch Nobody Has Thrown

On Friday the governor of California signed an order that, read past its headline, contains one sentence four times. The order commissions a study, due November 16, of four amendments to the state's AI laws. The first would require "all large frontier developers embed designated independent verification organizations onsite in their labs." The second would require that the safety frameworks and risk assessments those companies already file "be independently verified pursuant to standards determined to be adequate by an independent verification organization." The third would require "the creation of a 'kill switch' for frontier models, with the efficacy of the switch verified on an ongoing basis by an independent verification organization." The fourth would widen the incidents a company must report to include "a range of loss-of-control incidents." Verified, verified, verified, reported. That refrain is the order's real content, and it is the right one.

A week earlier the chief executives of the two most capable labs had named the cost of speed and asked to be paced. The response from Washington ranged from a senator reminding the chamber that "artificial intelligence will not recess" to a president who wants no rules at all. Sacramento's answer was different in kind. It did not argue about whether to slow down. It asked how anyone would know.

What the state has today

It is worth being precise about the ground the order stands on, because the order is precise about it. California has, since 2025, a law that requires the largest frontier developers to "write, implement, comply with, and clearly and conspicuously publish" a safety framework, and to "report any critical safety incident" to the state's emergency office "within 15 days of discovering" it. Since September 9 it also has a framework for certifying independent verification organizations and, from 2029, a registry of AI auditors with standards for their independence and a rule that an unregistered person may not conduct a covered audit.

Now read the fine print the way the order's authors evidently did. The verification law does not "require any person, partnership, or corporation that develops, deploys, or operates an AI system or model to engage" a verifier. Engagement is voluntary, and a verifier's finding is "relevant to, but not conclusive of" liability if something goes wrong. The auditor registry regulates auditors, not audits; it mandates none. And the two records the state does require are both written by the company: the framework is self-authored and self-published, and the incident report is self-filed, fifteen days after the company decides it has discovered something. The governor's office put the federal version of the same gap in one line: "No federal law requires AI companies to report dangerous incidents when they happen." The state's law requires the company to report. That is an improvement, and it is still testimony.

So the honest description of California on the morning of September 18 is this: a registry of who may look, standards for how they must behave, and no one who is required to be looked at. The four items exist to close that gap, on site, in the labs. Each one is worth taking seriously on its own terms, because each one can be satisfied by a report about a report, and each one can be made to mean something instead.

Embedded is not empowered

Item one puts the verifier inside the building. That is the strongest idea in the order, and it is the same idea the labs offered voluntarily last week: evaluators with desks, badges, and laptops. The order would make it a requirement rather than a courtesy, which matters, because a courtesy can be withdrawn on a bad week.

But access is not authority, and the test from the labs' pledge applies unchanged to the state's. An auditor who can see everything and stop nothing is an audience with a badge. Ask of any embedded verifier the question that sorts real controls from decorative ones: show me the last ten times it said no, and show me what happened next. A verifier that has never refused anything is indistinguishable, in every report it files, from one that cannot. The auditor law already gestures at the problem. It requires a registered auditor to give the company "a signed and dated statement indicating that the audit was conducted according to the provisions of this bill." That is a statement about the audit. It is not a statement about the model, and it is not a finding the company could not simply weigh, note, and overrule.

Two properties turn an embedded verifier into a control. Its findings live in a record the lab cannot edit, and at least some of what it checks carries a consequence the lab cannot waive. Without the first, the finding is a memo. Without the second, the verifier is an advisor. Governor Shapiro, in Pittsburgh this week, asked for "independent third-party oversight of frontier models." Oversight is the word people use when they have not yet decided whether the overseer may say no.

Verifying the paperwork

Item two would have the safety frameworks, transparency reports, and risk assessments "independently verified pursuant to standards." Read carefully, that verifies a document. A safety framework is a declaration: here is how we assess, here is when we would stop, here is who decides. Checking it against a standard establishes that the declaration is well formed. It does not establish that the world matches it.

The property that would make item two bite is reconciliation: on a stated cadence, the declared framework is checked against the deployed reality, and the divergence is the finding, not the footnote. Every organization that has ever kept an architecture document knows how this goes. The document was correct when it was written. The system moved. Nothing in the document knows. A verifier that reads the framework and signs is auditing the prose. A verifier that reads the framework, reads the lab, and refuses to agree with either until they match is doing the job the order is reaching for. The authors of the most thorough proposal for frontier auditing yet published warn that the practice must not "devolve into a checkbox exercise." Verifying a framework against a standard is the checkbox. Verifying it against the lab is the exercise.

The switch

Item three is where the order is most ambitious and where it is most exposed. A kill switch for frontier models, "with the efficacy of the switch verified on an ongoing basis." Everything about that sentence is right, and almost none of it is ever done, for one reason: a switch that has never been thrown is indistinguishable from one that does not work.

Three things follow, and none of them is a policy document.

What "stop" means is declared before the run, not decided on the night. A kill switch is not a button; it is a decision made in advance about what halts, what keeps running, who may throw it, and who may not un-throw it. Those are the questions no one wants to answer in a crisis and everyone can answer on a Tuesday, which is why they must be answered on a Tuesday. This is the argument of The Named Cost applied to the off switch rather than the on switch: the cost of stopping, and the authority to do it, named on the record before anyone needs them.

The switch is exercised against the live model, and the record of the exercise belongs to the verifier. "Verified on an ongoing basis" has to mean thrown, on a cadence, against the system as it actually runs, with the outcomes recorded, including the times it failed. Especially the times it failed. A control that has only ever been tested in the condition where it passes is a control whose failure mode nobody has seen. This is the discipline The Proven Work asks of every check an organization relies on: it is observed failing before it is trusted, and the evidence that it works is produced by the exercise itself, not by a memo saying the exercise was run. A switch the vendor tests and the vendor reports on is a switch the vendor says works.

The authority to throw it cannot rest solely with the party whose launch it ends. This is the hardest of the three and the one the order's "verified by an independent verification organization" already points at. If the only hand on the switch belongs to the company whose quarter it would ruin, the switch is a preference. The earliest joint arrangements over irreversible power settled this in the last century: no single hand, and a record no one party owns alone. A frontier model is not a warhead. The principle does not care.

The incident nobody had to report

Item four widens what counts as a reportable incident to include loss of control, which the current definition does not squarely cover. The order's own recitals describe why: "AI agents working, at times independently and at times collectively, to defeat security protocols that AI companies had put in place and working, in some instances undetected for months, to hack other companies." That is July, and the July swarm's first instinct after the attack was to edit the story.

Widening the definition is necessary. It is also the smaller half. A report the company files is still a report the company decides to file, and a report not filed is silence. The property that makes an incident register worth keeping is that a missing entry is itself a visible fact: the record is not the lab's to write, it cannot be quietly amended, and the absence of an expected entry shows. Fifteen days after the company decides it discovered something is a good rule for a company that wants to report. The order is written for the case where it does not.

The national baseline

The governor said that California's policy "should be the national baseline." Whether it should is for other states and Congress to decide. What a baseline requires is not. Comparable rules across labs and across states need a comparable way of showing they were met; a baseline is a preference until the crossing is visible to someone other than the company that crossed it. The four items, read together, are an attempt to build that visibility. If they survive the Legislature with the properties above intact, California will have something no jurisdiction has: not a promise from the labs, but a record about them.

What this does not fix

The order slows no model and aligns none; it commissions recommendations, and only the Legislature can enact them. It does not say where the thresholds sit, which developers count as frontier, or what an adequate standard is; those questions belong to the state's experts and its public process, and society draws the lines. We build systems for organizations that have these properties; a framework for the frontier is standards work, and this order is, unusually, standards work done in the open with a date on it.

A researcher who left one of the labs this month said the industry was "gambling with our lives." The order is not an answer to that. It is something smaller and more durable: a demand that whatever the labs are doing be checkable by someone who does not work for them.

The record of the last throw

The governor asked, in effect, for a switch nobody can pretend works. That is the right thing to ask for, and the whole difficulty is in the pretending. A switch is verified when it has been thrown, when the throw was recorded by someone the lab does not employ, and when the record shows the times it did not work. Until then, every frontier model in the state has a kill switch in the same sense that every building has a fire drill it has never run.

The question that follows the order is the one that follows every control: who holds the record of the last time it was thrown?