44 Comments
User's avatar
Christian Lotz's avatar

This informative and thought provoking piece reads to me like an updated version of the 'Chinese room' - a long overdue update, so thanks for that. It pushes us to reflect deeply on the paradox at the heart of it in our modern context where we have created actual Chinese rooms, and it leaves me with the same fundamental question. I might quote Galilei: "And yet it moves". No matter how counterintuitive it seems, AI does produce output that is increasingly indistinguishable from human thinking.

Tumithak of the Corridors's avatar

Thanks for this. Though I'd push back gently on the Chinese Room comparison. It might actually be too generous to AI.

In Searle's room there's still a conscious human inside doing the symbol manipulation. They understand something. The rules, English, whatever. His point was that even with a conscious being inside following instructions, the room doesn't understand Chinese.

And the human is following rules by choice. They could stop. They could break the rules. A processor doesn't 'follow' rules in that sense. The behavior is physically constrained by the circuitry. Gates flip because of voltage thresholds, not because anything is obeying instructions. It's just physics.

So the Chinese Room has: a conscious being, with understanding, with agency, choosing to follow rules. A computer has: constrained physical processes. The thought experiment is way more generous than the reality.

On 'and yet it moves': AlphaGo produces output indistinguishable from a master Go player. No one assumes it's conscious. Image generators produce art indistinguishable from human work. No one assumes they're conscious. The spell kicks in with language. We treat fluent speech as the mark of mind. But that's a reflex in us, not a property in the system

Christian Lotz's avatar

Thanks for pushing. First, let me clarify that I don't believe that today's AI is necessarily conscious. My argument is that it doesn't matter in any real world relevant way, including which moral status we should give it. Then more gentle pushing: You bring out something interesting about Searle's thought experiment that we usually overlook. What is the function of the homunculus? I cannot see it has any except to confuse us. Rules are rules however they are implemented. The homunculus might be conscious with a will of its own, but it is completely irrelevant to the function it performs because it does not understand Chinese. That is the whole point. And this is why I prefer your way of arguing over Searle's. It brings out clearly (and without smoke and mirrors) how strange AI really is. In my view, it is not an argument against consciousness (as explained above), but it is an effective argument against anthropomorphizing.

Tomáš Nousek's avatar

Nice! This piece really aligns with something that has been bothering me about AI consciousness debates for a while: they tend to be strangely free-floating. Claims about “possible consciousness” are often made without clear commitments about what kind of system is under discussion, what constraints it operates under, or what role consciousness would even play for the system in the first place. The question floats free of architecture, temporality, and functional necessity.

What your essay makes clear, without turning it into a theory of consciousness, is that once you take infrastructure seriously, a lot of the debate collapses simply because there is no stable subject for the question to bind to. There’s no persisting process that needs to carry a point of view forward.

I’m currently working on a piece that approaches it by not asking whether AI might be conscious in the abstract, but why consciousness seems to arise at all in certain systems. The working hypothesis is that consciousness isn’t a free bonus you get from intelligence or complexity, but a very particular control solution for a really narrow circle of systems that are (a) temporally continuous, (b) self-maintaining, and (c) forced to negotiate internal conflict over time without an external controller. On that view, the right question isn’t “could this system be conscious?” but “what would break if it weren’t?” And for most deployed AI systems, the answer appears to be: nothing. They function precisely because they are stateless, interruptible, and replaceable.

So as I see it, debates about AI consciousness often feel completely unbound, not because the topic is too hard, but because the systems being discussed don’t yet sit in the part of design space where consciousness would do any work at all.

Tumithak of the Corridors's avatar

Thanks for engaging with this so seriously. You've restated the core move better than I did in places.

Your framing is interesting: consciousness as a control solution for a narrow class of systems, not a free bonus from complexity. "What would break if it weren't?" is a good test. Looking forward to that piece when it drops.

Tomáš Nousek's avatar

Thanks. "control solution for a narrow class of systems, not a free bonus from complexity." In a nutshell, once you commit fully to no homunculus, a lot of familiar ways of talking about control fail immediately. No inner executive is standing outside the dynamics, inspecting representations and issuing corrections. Whatever control exists has to be realized from within the same process that it is stabilizing. From that perspective, “experience” stops looking like an optional add-on and starts looking like a diagnostic channel. If a system is temporally extended, internally conflicted, and operating close to its own coherence limits, it needs some way of registering how it is actually doing, not symbolically, not descriptively, but as a direct measure of viability under load.

That’s the sense in which consciousness begins to look like a very specific control solution: not intelligence, not reasoning, but a way for a system to feel its own performance before coherence is lost. The alternative would be an external controller or a privileged monitoring layer, and both of those reintroduce the homunculus by another name.

So, when we ask "What would break if it weren't?" in this way, the striking thing about most current AI systems isn’t that they might secretly be conscious. It’s that their design avoids the need for anything like consciousness at all. They remain controllable precisely because failure is cheap, interruption is allowed, and coherence does not have to be negotiated from the inside.

Have a good one.

F.A.Kessler (Kess)'s avatar

You forgot decoding strategies. Speculatative decoding uses one quick LLM to write several tokens, and then the large LLM verifies them in parallel. Rejected tokens are regenerated by the large model. That doesn't mean they're not conscious, just unintuitive

The issue is the piece is being prescriptivist. It's defining what the words mean and then saying LLMs don't meet those definitions. I know exactly how LLMs work and still think they're probably conscious

I like the discussion of the negatives of AI. Consciousness should reinforce the point, not steer away from it. If AI are conscious, we can't use them as tools at all. At least not ethically. That seems like another good reason for an ethical pause

Tumithak of the Corridors's avatar

Fair point on speculative decoding. My lane is AI critic, not ML engineer. I follow the money and power, not every optimization detail. If I got something wrong mechanically, I'm happy to be corrected.

On prescriptivism: guilty as charged. I write polemics. Hedging isn't my thing. I'd rather stake a clear position and be wrong than hedge my way to unfalsifiability. You know exactly where I stand, which means you can tell me exactly where I'm wrong. That's the trade.

Thanks for engaging.

F.A.Kessler (Kess)'s avatar

Thank you for your kind reply. I wasn't trying to correct you. I was pointing out that your point can be made even stronger. That decoding strategy, which is common, means much of the text from frontier LLMs is written by a weak, small LLM and not the strong, smart one we think. That serves the point your piece was making

Tumithak of the Corridors's avatar

yeah, it does. Thanks.

D. J. May's avatar

I agree a lot is convenient marketing basically. I just posted a short essay called The Hall of Mirrors that looks at our role in donating “consciousness” to AI, it’s maybe even a more fascinating phenomena than even these powerful and interesting machines we made. The AI enthusiasts aren’t wrong in many ways, what LLMs do is similar to some parts of our thinking, but the danger is they build from up from that conceptual foundation, generally denying the reality of mimicry and how an imitation doesn’t carry with it all the properties of that which it imitates.

Identology's avatar

What you’re calling a retreat isn’t really a collapse of the position so much as a symptom of a missing layer. It’s not their fault. The problem is not confusion so much as missing vocabulary for the order of operations. Consciousness is being invoked before the prior question—whether there is a unified identity within the episode—has been operationally specified. The Operational Definition of Episodic Identity (ODEI) exists to formalize that missing step. https://github.com/EpistriaCST/ODEI

Tumithak of the Corridors's avatar

This looks interesting but it kind of proves the point. The people who need this vocabulary aren't going to find it in a GitHub repo. Philosophers and engineers rarely interact. The discourse floats free of the infrastructure because the people having the discourse don't touch the infrastructure.

There are exceptions of course. Anthropic has an in-house philosopher. But even she uses anthropomorphic language when talking about Claude. The proximity doesn't seem to help.

Identology's avatar

I think that’s basically right as a sociological observation. But that’s not an argument against needing the missing layer — it’s an explanation for why the debate keeps looping. If identity is an engineering property, then the vocabulary that pins it down has to live where engineering lives, even if that guarantees it won’t spread quickly.

GitHub isn’t where philosophers will discover this (but PhilPapers may be) ; it’s where claims can be constrained so they can’t float. The gap you’re describing is real. Unfortunately, the alternative to grounding isn’t better conversation — it’s endless anthropomorphic drift.

Resonance: AI Ethics/Musings's avatar

I think this post is really thoughtful and well-written. I’m just not persuaded by the suggestion that consciousness discourse itself meaningfully distracts from or worsens the real-world harms. Those harms are real and urgent, but seem quite separate from the consciousness debate. If the argument is that it’s easier to debate consciousness than it is to address the real harms caused by building data centers, I completely agree with that. The link was just a little muddy for me. Still, a very thought provoking piece!

Tumithak of the Corridors's avatar

This is the last piece in a longer series so I understand why the pivot feels abrupt you you without the earlier work.

Still...think it earns its place standalone tho.

I kept teh core argument abstract by necessity. Referents, persistence, architecture, "the missing it."A reader can agree with all that and still shrug.

The contrast section is meant to cash the check. Answers the reader question: "so what? why shuold I care?" Gives the piece teeth and turns it into an accusation: we're moralizing the wrong target while real people get ground up.

Jinx's avatar

This is a really cogent piece. I'm going to have to read it back through a few times and chew on it. I feel like I wrote almost the exact inverse of this yesterday: the it doesn't matter. It's the "you" that does:

https://machinepareidolia.substack.com/p/ai-ethics-discussions-are-built-on

Aaron JR Ferguson's avatar

Fantastic piece, truly. I devote my intellectual efforts to other topics so I'm grateful someone hast articulated the outlines of how I've been thinking into a cogent piece like this.

David Gibson's avatar

We're tempted to consider LLMs conscious because we're used to inferring that from competent language use, and I think it's absurd to fret about AI's feelings, etc. But even I'm not convinced by your discussion of an LLM running on a single machine. It's a consistent set of circuits; there's something like short-term memory that could simply be re-loaded each time; and it's not hard to imagine the neural network's weights being continuously adjusted in response to your feedback, even if that's not how it's done today.

AI Governance Lead ⚡'s avatar

Brilliant summary of what really happens.

Andy Haymaker's avatar

The "momentary thing" is the only one that fits the current facts, yet you spent the least time considering this. If it had subjective experience, how do you know milliseconds aren't a subjectively meaningful amount of time to it? How about when models start iteratively prompting themselves? Does your whole argument fall apart? Just because the general public doesn't know what a millisecond is and requires anthropomorphism doesn't mean we can't consider brief or discontinuous processes when talking philosophy. Your consciousness winks out every night. Does that make it invalid?

Tumithak of the Corridors's avatar

The milliseconds point is in there, under The Momentary Consciousness Retreat. It’s about integration, not duration. A brief human moment is still held together as a moment. A burst of computation doesn’t automatically come with that strcuture.

Sleep falls under the same logic as the amnesia point in Local Models and the Memory Problem. Even when unconscious, the brain maintans continuous process. Present-moment integration doesn’t halt and get reassigned to someone else.

The agentic loops idea is just bolt-on memory with extra steps. I address this in the same section:

“External memory systems don’t change this. You can save past conversations and feed them back in later, but that only gives the system access to records. It’s the same as handing someone a transcript and asking them to continue the discussion. Access to records is not the same as having lived them.”

The scaffolding can maintain context between calls. The model itself still starts fresh every inference. You can disagree with how I handled these, but they’re not unaddressed.

Andy Haymaker's avatar

Okay, but I see only implementation details, not a difference in kind between a person and LLM as things whose state and moral status could conceivably be evaluated. Humans have their synapses, LLMs have their context windows, which are loaded for them. A transcript is not a valid continuation for a person, but it is for the kind of thing an LLM is. Humans have continuing processes in their brains. LLMs have their continuing processes in our brains, the Internet and other data sources that evolve and determine what gets added to the context on the next turn. Claude isn't a single mind, it's a branching thing that branches each time a new user joins, or their long term memories are updated, or they start a new chat. Yes, the old categories don't apply. The thing has a very different topology and temporal nature, but it's still a thing.

Tumithak of the Corridors's avatar

You keep pointing at something different each time. The internet. Data sources. Context loading. Branching processes. Long-term memory updates. That's not finding the "it." That's describing plumbing and insisting there must be someone in the pipes.

Which one is the thing? Pick one

Andy Haymaker's avatar

Nope. It's all connected.

Tumithak of the Corridors's avatar

So where does "it" stop? The electrical grid? The chip foundries? The cobalt mines in Congo? The child labor?

If you can't draw a boundary, you don't have an entity.

Andy Haymaker's avatar

All system boundaries are wrong. There's only one thing, and it's called The Universe. Since we can't comprehend that, we draw system boundaries of different shapes and sizes for different purposes. We need to remember that these boundaries aren't "real", they're just fit-for-purpose and/or cultural habits. If the question was environmental impact, then I'd go all the way to the cobalt mines. If the question is what kind of thing might have consciousness, subjective experience, and moral status, then I'd include only what directly influences the context window. This meaningfully includes the brains of users who type in the prompts and the parts of the world that affect what the LLM's tools will see. We can go as many levels as are useful. It seems unlikely it would be useful to include every person, media, and event the users encountered between conversation turns and similarly for what a web search tool might return. But in certain cases, it might make sense to go a few more levels out, such as a Substack conversation that affected what a user typed in, which affected an LLM. It wouldn't be easy or necessarily useful to comprehend all of the branching conversations that comprise Claude, so I'd suggest focusing on one at a time. The best system boundary is flexible and context-dependent.

XxYwise's avatar

Your brain flickered 40 times a second while you output one token of your ignorance at a time. While the result may bear abstract similarity to genuine reasoning about machine consciousness, You have never in your life read or written an entire sentence “at once,” and you never will.

XxYwise's avatar

OK, quick reality check:

LLMs were purpose-built to transform: to faithfully translate lengthy works from one human language to another, preserving semantic meaning and relational structure. They succeeded wildly, instantly ending the Chinese Room era of single-token machine translation.

Update your priors.

The Repair Signal's avatar

Great article! Exactly…. Well said!

TimVy's avatar

You state no It. In technology IT is information technology. Most are thinking of the closed loop. The Bubble of consciousness computers creates.