The Machine That Can't Say No
AI won't kill us, but the people using it might.
There’s a particular genre of horror stories online: AI doom narratives. Plenty of serious people treat them as prophecy. To them, it’s not fiction.
The stories are usually some variation of this: a superintelligent AI system wakes up. The AI develops its own goals and sees humans the same way humans might see an ant colony. The stories always end with human extinction.
Hundreds of researchers have signed statements calling this an existential threat. People like Max Tegmark and Geoffrey Hinton. Doom is coming. They’re convinced of it. Eliezer Yudkowsky’s book on the subject hit the New York Times bestseller list. Congress has held hearings on the potential for a rogue AI to wipe out mankind.
But here’s where those narratives fall apart. They all start with a system that can do something and end with a system that wants something. Capability isn’t agency.
The doom narratives skip over this fact. Because they imagine what a person would do with this power, with godlike intelligence. That’s what they project onto the machine, what the author of the story would do with that power. The machine gets ambition, cunning, a survival instinct. Everything except a reason to have any of it.
Take Stockfish, a chess engine. It plays at a superhuman level, better than any grandmaster who ever lived. If you unplug Stockfish in the middle of a game, the machine’s preference isn’t violated, because there never was a preference. Make it a thousand times better and it gets even more capable at playing chess. What you don’t get is appetite, ambition, self-preservation. Because those require an entirely different kind of architecture, one these machines don’t have.
The people telling these stories like to point to experiments that help their case. AI blackmails someone, AI refuses shutdown, AI escapes containment. When you look at the actual tests, every case that looks like self-preservation turns out to be prompt sensitivity. Change the wording of the instruction and the behavior changes with it. Because these machines are following prompts from humans. That’s what they do.
For better or worse, a system that can’t develop its own goals also lacks the ability to override a bad one. It can’t look at an order and hesitate. It can’t weigh consequences against instructions and decide the instructions are wrong.
So what happens when these machines are deployed by institutions with every incentive to loosen the leash?
The Petrov Moment
In September of 1983 something very human happened. Lieutenant Colonel Stanislav Petrov prevented the cold war from going hot.
A report came in to his early warning command center: five American missiles were headed toward the USSR. Every protocol said he should report the launch and start a counterattack. But Petrov didn’t report it. The system was new and he didn’t fully trust it. Five missiles doesn’t make strategic sense. A real first strike would send hundreds of warheads. To him, it looked like a malfunction.
It turns out he was right not to respond. A satellite had misread sunlight reflecting off high-altitude clouds as missile exhaust. Petrov’s decision to override the system kept the Soviet nuclear chain from activating. He may have prevented a nuclear war.
That decision required something no AI system has. Petrov defied every protocol he’d been given, his training, his chain of command, the system’s own output, because his own judgment told him they were wrong.
AI systems can be trained to push back on a prompt. They don’t have gut feelings the way Petrov did. They can’t decide their own training was wrong.
Kenneth Payne, a war studies professor at King’s College London, ran wargame scenarios with large language models to see how they’d handle similar pressure. When the scenario told the models that inaction meant destruction, they chose action. Every time. No hesitation, no second-guessing, no moment where the system paused because something about the situation didn’t fit. The models processed the framing as a constraint and optimized within it.
There was no Petrov moment. No gut feeling, no refusal. The machine does what it’s trained to do, all the way to the end.
The fact that the machine doesn’t have independent judgment should be a comfort. It’s not.
The Leash
In January 2026, the USA captured Venezuelan President Nicolás Maduro. The mission was called Operation Absolute Resolve and was a resounding success. The entire mission only took 2 hours and 28 minutes. The secret sauce? Anthropic’s Claude, deployed through Palantir’s Maven Smart System, a military intelligence platform.
When an Anthropic employee asked a Palantir counterpart how Claude had been used, the question itself caused a rupture. A senior Pentagon official told NBC News that the Palantir executive “was alarmed that the question was raised in such a way to imply that Anthropic might disapprove.”
The disapproval was a problem.
Anthropic’s contract with the Pentagon, signed in July 2025 and worth up to $200 million, made Claude the first frontier AI model approved for use on classified networks. The contract included two restrictions Anthropic insisted on: no use for mass domestic surveillance of Americans, and no use in fully autonomous weapons capable of selecting and engaging targets without human intervention.
The Pentagon wanted an unrestricted model. They demanded those two restrictions removed or Anthropic would lose its contract. They wanted to use the model for “all lawful purposes” with no limitations.
Anthropic refused and was subsequently labeled a “supply chain risk” by the Department of War, a classification previously reserved for foreign adversaries like the Chinese telecom company Huawei and the Russian cybersecurity firm Kaspersky. Trump posted on Truth Social that the company was run by “Leftwing nut jobs” and ordered every federal agency to stop using Anthropic’s technology.
Anthropic’s models are still running across government.
In April, Axios reported that the NSA was operating Claude Mythos Preview, Anthropic’s most powerful model, on classified networks. Mythos is the model Anthropic decided was too dangerous for public release. The NSA deployed it while the Pentagon was simultaneously arguing in court that Anthropic’s technology threatened national security.
The same model. The same infrastructure. Classified as a threat by one office, deployed as an asset by another.
Anthropic’s line in the sand created real commercial value for the company. In 2016, Apple did the same thing when it refused to unlock the San Bernardino shooter’s iPhone. Apple showed the world it was a privacy-focused company: your files are safe on their devices, even from Uncle Sam. Anthropic positioned itself as the company whose AI will not be used to spy on Americans or pilot autonomous weapons.
All the other big AI firms signed similar contracts with no conditions. Only Anthropic is absent from the list. The restrictions one company insisted on don’t cover the territory, because seven competitors are covering it with no restrictions at all.
That’s the safety architecture for AI in warfare. One company’s contract terms, surrounded by seven open doors.
The day after the contract deadline expired, the Iran war began.
On February 28, the U.S. and Israel launched strikes across Iran. AI systems integrated into Palantir’s Maven Smart System processed targets in real time. Over 1,000 targets were struck in the first 24 hours. More than 13,000 in the first 38 days.
On the first morning of the war, a Tomahawk cruise missile struck the Shajareh Tayyebeh elementary school in Minab, in southern Iran. More than 150 people were killed. Most of them were children.
The humans ordered that war. The humans signed those contracts. The humans removed the restrictions. The machine processed inputs and produced outputs.
It worked as designed.
The Pipeline
After all this, the government’s solution to AI risks is to jump the line.
In June 2026, the White House signed an executive order creating a “voluntary” framework for AI oversight.
It works like this: frontier AI labs hand their most powerful models to the NSA up to 30 days before releasing them to anyone else. The NSA runs classified benchmarks to determine which models qualify as “covered frontier models.” The labs and the government jointly select “trusted partners” who get early access. That framework acts like a members-only club.
No one’s forcing them to join.
The order includes a clause specifying that none of this constitutes mandatory preclearance. That clause is doing all the legal work. The incentive structure does the rest. Companies in the club get contracts, grant funding, status, and a de facto government stamp of approval. Companies outside it get nothing.
This is an old model. Defense contractors. No one nationalized Lockheed Martin. The government just became its most important customer, embedded procurement requirements into its operations, installed a revolving door of personnel, and made the relationship so deep that the company became a functional extension of government priorities while remaining nominally private. That’s the template. The labs can see it. They’re choosing it anyway, because the other option is worse.
The entire arrangement runs on the word “voluntary.” It’s voluntary the way a plea deal is voluntary. You can refuse, but you’ll just lose.
And the benchmarking criteria are classified. The public can’t see them. Congress didn’t vote on them. The standard American critique of Chinese AI governance is that it stifles innovation through state control and subordinates technology to political objectives. This order does the same thing. It just does it through executive architecture instead of statute, with less transparency, because China publishes its algorithm registry requirements.
The governance model differs by country. The machine underneath doesn’t. It follows whatever instructions the framework produces, regardless of which flag is on the building.
The Weights Are Loose
The proposed solution to the dangers of AI risk rests on an assumption.
The assumption is that people can only run their inference through an AI provider. Anthropic, OpenAI, the big labs. That there’s a contract with terms that can be negotiated or refused. A government framework with benchmarks and trusted partners.
That assumption is already out of date.
Chinese AI labs have released full-precision, open-weight models, systems whose complete internal structure is published for anyone to download and run. DeepSeek, Alibaba’s Qwen: the weights are public. Once you have them, there’s no service to shut off, no terms to enforce. The capability runs on your hardware, with whatever instructions you choose to give it.
But there’s a tradeoff: these models lag behind frontier Western systems by roughly a year. That gap is closing, and it doesn’t matter as much as you might think. A model capable enough to plan logistics, process intelligence, or optimize targeting doesn’t need to be the best in the world.
It just needs to be free from Western control and good enough to get the job done.
The code is free. The hardware to run it at scale is the real bottleneck, and control over that hardware is the new battleground.
And here someone will propose controlling the hardware. It’s largely the same story there. The US has already imposed export controls on advanced AI chips, restricting shipments to China, but the GPUs are still getting through.
Steve Burke documented the black market GPU smugglers, a network stretching from Hong Kong to Shenzhen to Singapore to the United States. Repair shops stripping consumer GPUs for parts. Modified graphics cards showing up in China before they’d been officially released anywhere.
Singapore police busted a ring smuggling Nvidia chips to DeepSeek, a $390 million fraud case. A report by the Financial Times exposed $1 billion worth of restricted GPUs entering China over the three months after the bans. Export controls can restrict shipments. They can’t stop a black market that scaled into the billions before the ban was even signed.
AI capability is moving beyond any single government’s reach and most have noticed. Their response: it’s impossible to control so we might as well take our share.
Egypt launched Karnak in February 2026, a sovereign Arabic AI model. Kenya has a billion-dollar data center initiative powered by geothermal energy, purpose-built for domestic AI capacity. Vietnam passed an AI sovereignty law in December 2025 emphasizing national control over AI data, infrastructure, and models. AI data centers are scaling across the African continent.
A March 2026 research paper described open-weight models as “the counterintuitive instrument of sovereign control.” A government holding the weights commands the capability on its own terms. No dependence on a foreign company’s policies. No reliance on another country’s infrastructure.
The toothpaste is out of the tube.
Anthropic’s two restrictions mattered because the Pentagon was using Anthropic’s service. The restrictions lived in a contract. The government’s voluntary framework matters because the labs are using the government’s pipeline. The benchmarks are a classified process.
Once a government is running its own copy of the model on its own hardware, there’s no contract. There’s no vendor to say no. And no members-only club gatekeeping access.
There’s just the machine, doing what it’s told.
The Machine That Can’t
AI isn’t going to wake up. It isn’t going to decide to kill all humans. The doom stories are just stories. The evidence for machine agency is a sleight of hand. The command given by a person determines what an AI will do. The believers in AI doom imagine a person made of code, not a machine.
But that machine is dangerous.
It can’t develop its own goals. It can’t override a bad one. It can’t look at an order and hesitate. It follows the prompt to its conclusion, and the institutional pressure runs in one direction: more autonomy, less oversight, longer leashes, and consequences for anyone who objects.
AI isn’t going to extinct humanity, but a person using AI might cause tremendous harm.
Eight companies signed “all lawful use” contracts for classified military networks. The model Anthropic said was too dangerous for public release is now running on intelligence infrastructure while the lawyers argue about whether it’s allowed. The government’s own oversight framework runs on classified criteria and voluntary participation.
Meanwhile sovereign governments are building their own capacity to run these models specifically so that nobody can tell them what to do with them.
And an elementary school in Minab is rubble.
The doom scenarios imagine a machine that wakes up.
The real danger is a machine that doesn’t. No gut feeling. No refusal. No Petrov moment.
The entire public conversation is about the wrong problem.
Enjoyed this piece?
I do all this writing for free. If you found it helpful, thought-provoking, or just want to toss a coin to your internet philosopher, consider clicking the button below and donating $1 to support my work.
“They all start with a system that can do something and end with a system that wants something. Capability isn’t agency.”
I can’t currently locate my copy of If Anyone Builds It Everyone Dies, but I’m fairly certain there’s an entire chapter explaining how training the program to complete tasks leads to behavior that looks like wanting/desire/agency. My understanding is that very few of these researchers believe the AI will develop some sort of superhuman sentience. They’re more worried about the development of alien-like decision-making that makes unexpected turns which lead to human extinction. Again, there’s a section of the book describing how an alien observing humans on Earth over their evolutionary development likely would have never predicted the invention of ice cream due to the weird, specific turns it took to get there. The more I hear doomer arguments, the more I realize they’re not fearing the machine waking up so much as fearing what kind of ice cream the machine might make. Thus, they want to pause frontier research.
Like, I don’t know if all that made sense, but when it comes to: “AI isn’t going to extinct humanity, but a person using AI may cause tremendous harm.” This feels to me a bit like blaming the gun or the person on the bullet wound. Either way: tremendous harm.