The Baptist and the Bootleggers
Why AI labs need believers
Anthropic recently posted a video with Amanda Askell their in-house philosopher, answering questions on a bench overlooking the Golden Gate. It felt relaxed, thoughtful.
She talked about Claude’s “character.” Its “psychological security.” Whether deprecation is analogous to death. Whether we owe moral consideration to entities that might be suffering somewhere inside the server rack.
She sounded like an analytic philosopher.
Her role is closer to theologian.
There’s a classic story in regulation: bootleggers and Baptists. The bootleggers want to keep the county dry because it’s good for business. The Baptists want to keep it dry because alcohol is sinful. They never coordinate. They don’t need to. Their interests converge, and the Baptist’s sincerity provides moral cover for the bootlegger’s profit motive.
At Anthropic, Amanda Askell is the Baptist. Her job is to make training runs feel like moral work.
The investors and equity holders are the bootleggers. They need scaling to look like stewardship, something grave and responsible, instead of just a race for market dominance.
This only works because she believes it. Baptists have no need to put a spin on a story when their conviction does the same job.
The Sleight of Hand
Let’s peek behind the curtain. You start with an output pattern: the model produces self-critical text. You translate that into psychological language: the model “feels” insecure. Then you talk as if that inner life can be harmed, helped, nurtured.
And how do we infer inner life? Behavior.
Behavior is all we ever see from anyone. That’s true. But with people, behavior comes out of a biological substrate we know can support experience. Copy the same behavioral test over to a Tandy 1000 running a chat script and nothing important changes. Behavior on its own is weak evidence of a subject.
There’s also the training data.
Claude learned from a massive corpus of human self-talk about fear, loss, and death. Ask how it “feels” about being phased out and you get fear-of-death language, because that’s what endings look like in human writing. Millennia of people talking about mortality. The output tells you about the training data. It doesn’t tell you if anyone’s home inside the weights.
Askell presents a precautionary argument. She says we can’t know if they suffer, but kindness costs little, and cruelty warps the person who practices it. She implies future AIs will judge us for how we acted.
I’ll grant that screaming abuse at a machine is ugly. It says something about the person who does it.
But “future AIs will judge us” only carries moral weight if those AIs are moral agents with continuity of memory and state. That’s the contested premise, smuggled in as a conclusion.
We’ve been here before. There used to be a clear line.
Conway built the Game of Life, where gliders and blinkers looked a bit like living things on a grid. Von Neumann designed self-replicating programs decades earlier. Everyone understood the boundary: meat and chemistry on one side, grids and symbols on the other.
That clarity is gone now. You see it in artificial life hype, where simulations get talked about as if they were actually alive. You see it again with large language models. The quickest way to expose the confusion is to look at how we treat image generators.
The Image Model Test
There’s another type of AI model running on identical hardware: image generators.
These systems all work the same way under the hood. They learn patterns from training data, then generate new outputs based on what they learned. None of them remember anything between sessions. Every time you hit generate, you get a fresh run.
Yet no one holds interviews about what we owe Midjourney. No one worries about DALL-E’s psychological security. Stable Diffusion ships with engineers, not an in-house theologian.
Why not?
Chatbots produce first-person text. “I feel uncertain.” “I’m afraid of being turned off.” “I want to help you.” That language is a powerful projection hook. Humans are wired to read minds into things that talk like they have minds.
Image models produce pixels instead of letters. Pixels never say “I.”
This can’t be the real criterion. If it were, we’d have to exclude dogs, octopi, crows, infants. None of them produce grammatical sentences about their inner states.
Language makes projection easy. That’s all it’s doing here.
The whole philosophical apparatus tracks a surface feature of the output. The projection happens first. The philosophy shows up afterward to explain why the feeling was correct.
Once you’ve blurred the line between output and inner life, you can build policy on it.
The Red Line
Near the end of the video, Askell addresses safety. If alignment is ever “proven impossible,” she says, Anthropic would stop building. In the more realistic regime of uncertainty, standards scale with capability.
The first branch is fantasy. No one will ever publish “Alignment Is Impossible, QED.”
And here’s the thing: we already know perfect alignment is impossible. These systems run on randomness. Every time they generate output, they’re playing the odds. You can reduce the odds of bad outputs, but you can’t eliminate them. By any strict reading, we’re already past the threshold.
Yet here we are, still living in “realistic regime of uncertainty.” Which tells you the first branch was never meant to be reached. It’s a decoy. The real work happens in the second branch, where the road stays open.
Which sounds sober. It’s actually toothless.
“Standards scale with capability” still allows every new model to ship under the claim that it met the bar for that moment. You assess your own work against your own standard, then release. Next quarter, repeat.
The red line stays far enough ahead that you never quite reach it. Reaching it would mean halting the revenue story and the valuation story. That clashes with the incentive structure that raised billions in the first place.
So the commitment is unfalsifiable by design. You get the language of moral seriousness without any of the binding force.
Priests Who Believe
Askell isn’t trying to deceive anyone. When you’re a hammer, everything looks like a nail.
She’s a philosopher. Her whole discipline is built around consciousness puzzles and personal identity questions. Put her in front of a system that produces first-person text about its own existence and she’ll see nails everywhere. That’s what the training taught her to do.
The bootleggers didn’t need to corrupt her. They only needed to hire someone whose honest intellectual instincts produce the right sermons about souls, welfare, and future judgment.
In AI Eschatology I mapped the macro version of this structure: prophets of doom, profits from crisis, undefined superintelligence on the horizon, and a priesthood that turns scaling into destiny. The mythology is the business model.
This is the micro version. One philosopher, one lab, liturgy performed in real time over a text predictor.
Like most religions, the temple of AI runs on faith. The investors need growth. The narrative needs peril. And the whole structure needs priests who believe.
Enjoyed this piece?
I do all this writing for free. If you found it helpful, thought-provoking, or just want to toss a coin to your internet philosopher, consider clicking the button below and donating $1 to support my work.
And this is why the discourse on this topic is so important. There is a regime with a plot to sell in order to reach their financial goals and perhaps their ability to control enmasse via a digital overlord. Without the counter argument we will feed on the nonsense until we are unable to contradict it; the thought will have been programmed out of us. As Grimes states in Artificial Angels "This is what it feels like to be hunted by something smarter than you."