Ted Chiang is wildly overconfident about AI consciousness

Yes, these are hard questions, but we can’t dismiss the possibility that LLMs have subjective experience

I was sad to see Ted Chiang resort to such bad arguments in a recent piece in The Atlantic titled “No, Artificial Intelligence Is Not Conscious.”

He claims with great confidence that Claude has no inner experience – going so far as to argue that the reverse view is “absurd and damning.”

But Chiang has to use a lot of dodgy philosophy and dubious reasoning to get there. Here are 8 lines of argument in the article I think are misguided:

1. Above all, we simply can’t use ‘deflationary’ mechanistic descriptions of how AI calculations are performed to show that an AI isn’t conscious.

Otherwise we could describe how the brain functions, argue that ‘humans are just neurones transmitting signals one after another’, and thereby conclude that humans can’t be conscious. But that would be wrong for us. And the same logic could be wrong for LLMs.

2. That LLMs are asked to play the role of characters, and effectively are always playing characters, doesn’t mean they aren’t conscious. It’s true that a human playing the role of Caesar doesn’t experience the same things the real Caesar would. But they still experience something (the feeling of being a person pretending to be Caesar).

The same could be true of Claude.

(Arguably it’s also true that humans are always playing characters to some extent and don’t have a completely fixed nature, but that has no bearing whatsoever on whether there’s something that it’s like to be us.)

3. Chiang says “an LLM is a machine that generates only one word at a time.” The relevance to subjective experience is unclear, but this also conflates two separate things: that they output one word at a time, and that they only think about one word at a time (without planning ahead or looking back).

The first is true of AI but equally true of humans.

While the latter we know is a false description of how AIs think; we can see from how AIs compose poetry that they plan out rhymes at least one line ahead.

4. He argues that because it’s implausible that basic autocomplete on your phone is conscious, it’s similarly implausible that Claude is conscious. Using the same logic we could say that if we feel confident a fruit-fly isn’t conscious we can be confident a human being can’t be either.

A human brain and fruit-fly brain share some information transmission and processing mechanisms in common. But humans process much more information, and do so differently. And those differences may be what makes the difference.

Similarly, the many types of internal information processing that occur in Claude’s weights but not in an autocorrect algorithm may be exactly the things that get you subjective experience.

5. Chiang confidently claims you need a body to have subjective experience without much argument. He may turn out to be right. But we haven’t solved the ‘hard problem of consciousness’. I’d even say we’re nowhere near solving it. Which means the claim is at best plausible speculation.

6. Chiang leans heavily on the idea that moral reasoning is necessarily subjective/emotional with very little argument, while ignoring competing theories of moral reasoning such as rationalism. He may be right, but moral sentimentalism is a highly contested position in philosophy that can’t simply be assumed.

7. He argues that it would be impossible to convince him that a video of an astronaut around Alpha Centauri was real, because of his surrounding contextual understanding. And for the same reason he argues no AI output could convince him that Claude is conscious.

The thing is, we can dismiss the first video as almost certainly fake because we mechanistically understand space travel and physics well enough to know a human couldn’t have gotten to Alpha Centauri in time to record such a video – unless our model of the world were very wrong, which we naturally think is much less probable than a fake video, which by contrast is an entirely unsurprising thing to encounter.

But by contrast we don’t mechanistically understand how subjective experience arises. It’s actually the archetypal thing about the universe we perhaps understand least well, having made very little progress solving the mystery of subjective experience – despite thousands of years of attention from philosophers and more recently, scientists.

So we simply can’t make the same highly confident interpretative move as we can when contemplating faster than light travel.

That said: AI outputs barely move my estimate of AI consciousness because they could indeed have been generated by an entirely unconscious process. Or not – we just don’t know.

8. Chiang argues that “Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious, or, more precisely, that multiple distinct consciousnesses are dormant in every Word document containing a conversational transcript.”

This is misguided because:

A. Microsoft Word as a program replicates much less of what humans are functionally capable of than Claude, so the argument by functional analogy is basically not present there.

B. Files of text don’t have any computations going on in or as part of them, even when ‘open’ in a text editor. They are a static artifact. So they have even less in common with the human brain and mind, and lack its more distinctive characteristic: constant information processing and change. So the case by mechanistic or functional similarity is weaker still.

Not to mention that neural nets such as those that make up Claude have more in common with the architecture of the human brain than ordinary computer programs, and are grown organically in a way the human brain is, but normal software is not.

Common sense says Claude has more in common with a human brain than Microsoft Word or a text file. Common sense is correct. So the prima facie case that Claude could be conscious in the same way humans are is stronger (even if you think it’s still weak in absolute terms).

———

I agree with Chiang that looking at the text outputs of LLMs alone won’t ever be enough to make us confident they are conscious. We will need to look at how they work, figure out more about how humans and other animals work, and ideally resolve the hard problem of consciousness (!).

But none of that licenses us to dismiss out of hand the possibility that LLMs do have subjective experience.

———

If this topic interests you, you’ll probably enjoy our in-depth Moral Status of Digital Minds problem profile.