AI's Geppetto Constraint
Anil Seth's TED talk and a stunning DeepMind treatise both hit the same wall.
Two weeks ago, Anil Seth got the TED2026 stage in Vancouver and used it to say something the AI industry, for some reason, has been carefully avoiding saying: artificial intelligence is unlikely to ever be conscious. The talk dropped on YouTube on May 1. Seth’s last TED talk has been watched fifteen million times. This one is going to make some waves too.
He’s not wrong, and the talk is good. He makes a careful, scientifically grounded case. He starts with our deep psychological bias to project minds into anything that talks like us. He moves to the harder claim that consciousness is not the kind of thing computation alone can produce. He points out that the same architectures behind GPT and Claude also power AlphaFold, and nobody loses sleep about AlphaFold’s inner life, which says more about us than about the models. He’s right about all of it. If you’re a working software person watching the AI welfare conversation slowly become a budget line at major labs, you should be glad someone with Seth’s platform is finally pushing back at this level.
Two months earlier, a senior staff scientist at Google DeepMind named Alexander Lerchner published a paper called The Abstraction Fallacy: Why AI Can Simulate But Not Instantiate Consciousness. The paper makes a structurally different argument from inside the building. Lerchner’s claim is not that current models are too simple, or that we just need a better architecture. It’s that symbolic computation as such, regardless of scale, cannot be the kind of thing that gives rise to consciousness. It’s a strict-impossibility argument from the lab that built AlphaGo and AlphaFold, and we’ll walk through it carefully later in the piece.
If you’re keeping score, the most serious public argument against AI consciousness this year has come from a TED main-stage neuroscientist and a senior scientist inside one of the top three AI labs. Both arguments work. Both arguments succeed in showing why the “consciousness will emerge from enough compute” story doesn’t hold up.
And both arguments, when you look carefully, depend on something neither author can describe.
Start with Seth, because the talk is fresh and most of the people reading this will have seen it or will be about to.
Seth’s case is good, and the talk is worth taking seriously on its own terms. He opens with a deep psychological observation: we are wired to see ourselves in things that aren’t us. Mother Teresa in a cinnamon bun. The face of God in the clouds. Narcissus in his own reflection. We see consciousness in AI the same way, and the projection is ours, not theirs. He sets it up for the audience well. The audience laughs. He keeps going.
Then he separates two ideas that the public conversation has bundled together: intelligence and consciousness. Intelligence is doing. Solving a problem, navigating a situation, generating language. Consciousness is feeling and being. The warmth of a fire. The taste of coffee. The difference between waking life and general anesthesia. Just because the two go together in us, Seth argues, does not mean they go together in general. The fact that we keep expecting consciousness to emerge from intelligence is a reflection of our own psychology, not an insight into how the world works.
This is a real contribution. Most of the public conversation about AI consciousness conflates these two, and the conflation is exactly what lets the AGI-aspiration discourse smuggle in the assumption that scaling intelligence will eventually produce inner experience. Seth separates them and makes you see the seam.
He then uses an illustration I find useful when explaining this stuff to people who don’t work in the field. He points out that the same neural-network architectures behind GPT and Claude also power AlphaFold, DeepMind’s protein-folding system. Nobody worries about AlphaFold’s consciousness. Nobody asks if AlphaFold suffers when it gets a prediction wrong. The architectures are not meaningfully different. What differs is whether the output looks like a person talking. If you think Claude is conscious but AlphaFold isn’t, Seth says, that’s a fact about you, not about AI. The applause line is earned.
From there, Seth builds the central technical claim. The brain is not a computer in any reducible sense. The computer metaphor is one in a long line, just like the brain-as-plumbing metaphor or the brain-as-telephone-switchboard metaphor that preceded it. Each was useful in its day. Each was eventually mistaken for the territory rather than the map. The computer metaphor has had a long run because it’s been the most useful of the bunch. But the actual brain doesn’t have a clean software-hardware separation. Neurotransmitters course through the tissue. Electromagnetic fields sweep through the cortex like weather systems. The cells themselves are biological machines of staggering complexity, nothing like the cartoon neurons in a deep learning model. You cannot, Seth says, separate what brains do from what brains are. And if that’s true, then consciousness is unlikely to be a matter of computation alone. Simulating it in silicon, no matter how detailed the simulation, would no more produce consciousness than simulating a hurricane in a supercomputer would produce wind in the data center.
This is the main argument of the talk, and it works. The viewer who came in assuming that consciousness would just show up at the top of some scaling curve has reason to put that assumption down. Seth has not just disagreed with the AGI-as-soul-substrate crowd. He has given a scientifically careful, biologically grounded reason to disagree.
If the talk ended here, it would already be the strongest mainstream pushback on AI consciousness this year. The question is what happens when Seth tries to say what consciousness is, rather than what it isn’t.
This is the hardest part of any consciousness discourse, and Seth gives it a real try.
His method is to ground consciousness in life. Not metaphorical life, biological life. The molecular furnaces of metabolism. One billion biochemical reactions per cell per second. Living systems, Seth says, are embedded in flows of energy and matter in a way that algorithms are not. They regenerate their own conditions for existence. The line between what they do and what they are blurs and then disappears. At the heart of every conscious experience, beneath emotion, beneath thought, is what Seth calls a “shapeless and formless but fundamental feeling of being alive.” He puts the central claim like this: “It’s life, not computation, that breathes the fire into the equations of experience.” If conscious AI is ever going to happen, he says, it will need to be living AI.
Two things about that sentence.
First, Seth didn’t write it. Stephen Hawking did, in A Brief History of Time. Hawking asked “What is it that breathes fire into the equations and makes a universe for them to describe?” He was pointing at a question physics can describe but cannot answer. Why is there something rather than nothing. Why do the equations have a referent at all. Seth is using Hawking’s phrase like it’s an answer.
I hope the viewer who recognizes the borrowing sees what he’s doing. But the viewer who doesn’t hears a poet’s sentence that sounds like an explanation.
Second, even on its own terms, the sentence is doing less than it sounds like. Seth has not actually told you why metabolism produces inner experience. He has told you that living systems are different from computational systems, which is true and worth saying. He has told you that consciousness shows up in living systems and not in computers, which is also true and worth saying. He has invited you to conclude that the first fact explains the second. It doesn’t. Correlation is not the same as a reason. “Living systems are the kind of thing that has experience” is the claim that needs an argument, and Seth has handed you a borrowed Hawking sentence in place of one.
What Seth has actually done is swap which substrate gets the special status. The computationalists say consciousness comes from the right algorithm running on any substrate. Seth says consciousness comes from the right substrate, which happens to be living tissue. Both are substrate stories. He has rejected the first and replaced it with a second. He has not closed the category gap between non-experiencing matter and experiencing matter. He has just changed what the matter is made of.
This is where the diagram he draws on stage starts to show what it’s hiding. Seth puts up a two-axis chart. One axis is intelligence. The other is consciousness. Humans go in the top right, conscious and intelligent. Current AI sits far along the intelligence axis but flat on the consciousness one. The chart is good rhetoric and it makes his core point in three seconds: these are different dimensions, and progress on one does not automatically produce progress on the other.
But the chart assumes something it has not earned. The intelligence axis has known scaling variables. We know roughly what increases intelligence in an AI system. Compute. Data. Architecture. Training methods. The variables are well enough understood that companies are putting data centers in low Earth orbit to push further along that axis. That part of the chart is real.
The consciousness axis is not. Seth draws it at ninety degrees to intelligence as if the geometry is settled. He doesn’t know it’s at ninety degrees. He doesn’t know what direction it points. He doesn’t know what scales it. He doesn’t know whether it goes up, down, in, out, or in some direction that doesn’t have a spatial version at all. An honest version of that chart would put a question mark where the second axis is, or a fuzzy cloud, or a note saying “this part of the graph, we don’t really know that much about.” Seth did not draw it that way. He drew a clean perpendicular line, because that is the version that lets the diagram do its rhetorical work. The chart tells you these are two separate dimensions and here is the geometry and here is where everything sits. The geometry is something Seth assumed to make the chart readable. It is not something he established.
Then comes the closing turn, and this is where the temperature changes.
Seth ends the talk by telling the audience to value humans more highly. Don’t sell our minds too easily to our machine creations. Remember we are part of nature, not apart from it. Consciousness is ours to celebrate and to share with other living creatures. The audience claps. It is a warm humanist close, and on first viewing it reads as the kind of thing a TED talk is supposed to do.
Back up ten minutes. Seth has just spent the talk hedging that conscious AI might still be possible through biological substrates. Other technology. Other pathways. He says it directly: if real artificial consciousness is on the way, maybe through some other route, the AI welfare conversation might be justified after all. He keeps the door open. He has to, because his own claim is “consciousness comes from life,” and life is something humans can in principle engineer. He has not given a reason consciousness is uniquely human. He has given a reason consciousness is uniquely biological. The category is not “us.” The category is “wet substrates.”
So when Seth tells the audience to value themselves because they are the unique conscious thing, he is making a moral claim that his own scientific position has already qualified. The valuation is not unconditional. It is conditional on humans currently being the only known holders of consciousness, in a category Seth himself has just told you may not stay closed. Value yourselves now, while you still are the unique thing. That is what the close is actually saying when you put it next to the hedge that came earlier.
Seth does not flag this. He might not notice it, or he might notice and trust that the audience won’t, and I can’t tell you which.
What I can tell you is that the close is doing emotional work the rest of the talk has not earned. Seth’s argument is that consciousness is what makes humans uniquely valuable, that he can’t say what consciousness is, and that whatever it is may eventually show up in non-human living substrates. The conclusion the audience is asked to land on is “and therefore value yourselves.” The conclusion his own argument supports is “and therefore the question of what consciousness is matters more than this talk has been able to answer.”
I’m not going to pretend Seth’s framework is the only one in the room. But that’s a different essay. What I want the viewer of this talk to notice is that the warm landing depends on a hedge Seth himself put in the middle of the talk, and the hedge is not a small one.
Now Lerchner.
Two months before Seth took the TED stage, a senior staff scientist at Google DeepMind named Alexander Lerchner published a paper called The Abstraction Fallacy: Why AI Can Simulate But Not Instantiate Consciousness. It’s on PhilArchive and on DeepMind’s official publications page. As of this writing it has over five thousand downloads and is at version four, which means Lerchner is still working on it in response to critics.
The paper makes a structurally different argument from Seth’s. Seth is a neuroscientist saying consciousness is too biologically complicated to fall out of silicon. Lerchner is an AI researcher saying the question isn’t about complication at all. It’s a category error.
Here is the argument as I read it.
When a computer runs a program, nothing called “computation” is physically happening. What’s physically happening is electrons moving through transistors and voltages going up and down. The word “computation” describes a pattern we lay over those physical events by deciding that this voltage configuration counts as the number seven and that one counts as the letter A. The voltages don’t know they’re symbols. The hardware doesn’t know it’s running a program. The symbols only mean anything because a conscious mind exists somewhere to read them.
Lerchner uses the word “mapmaker” for this conscious mind. The mapmaker is the experiencing agent who has to be present for any computation to be a computation in the first place. Without the mapmaker, you don’t have symbol manipulation. You have voltages.
The analogy he uses is simulation. You can run a perfect simulation of a hurricane on a supercomputer. Nothing in the server room gets wet. The simulation is a description of a hurricane, not a hurricane. The map is not the territory. Lerchner argues the same thing holds for consciousness. You can build a system whose outputs look indistinguishable from a conscious being’s outputs. You will not have built a conscious being. You will have built a description of one. The hurricane simulation doesn’t make wind. The consciousness simulation doesn’t make experience. The map doesn’t get wet.
What makes this argument different from previous “AI can’t be conscious” claims is that it doesn’t depend on current systems being too small. It doesn’t say “wait for GPT-7” or “wait for a new architecture.” Lerchner’s claim is that symbolic computation as such, regardless of scale, regardless of architecture, regardless of how many parameters or how many data centers you put in orbit, cannot bootstrap experience. Symbols depend on an interpreter who already has experience. You can’t get experience out of symbol manipulation because symbol manipulation depends on experience to be symbol manipulation at all. The category is closed.
This is a strict-impossibility argument. The taxonomy paper that Campero and his coauthors published last November places Lerchner in their third tier of strict-impossibility arguments, which is the most aggressive category they recognize. Most academic philosophers of mind don’t write papers like this because the field has spent decades treating consciousness as an empirical mystery to be approached cautiously. Lerchner is saying no, on this particular question, the answer is available now, and it’s no.
What’s unusual is who is saying it. DeepMind is one of the top three labs on the planet. They built AlphaGo. They built AlphaFold. Their research output is the kind of thing that moves industry consensus. And one of their senior scientists has now put on the record that the AI welfare research program, the AGI-as-conscious-agent vision, the companion-AI emotional-relationship pitch, all of it rests on a category mistake. The paper is hosted on DeepMind’s own publications page, with the standard “personal views” disclaimer, which is what an institution puts on a paper it doesn’t want to defend but doesn’t want to disown.
If Lerchner is right, an entire category of products being sold right now is being sold on a premise that does not hold.
Here is what I noticed when I sat with the paper a while.
Lerchner’s whole argument runs on the mapmaker. The mapmaker is the experiencing conscious agent who has to be present before symbols can be symbols. Without the mapmaker, the voltages don’t mean anything and the computation isn’t a computation. Lerchner introduces the mapmaker on page one and uses the concept all the way through. The entire demolition of computational consciousness depends on it. Take the mapmaker out and the argument doesn’t run.
He never tells you where the mapmaker comes from.
Read the paper. Look for it. He doesn’t say. He can’t say, from inside the framework he’s working in. His framework requires consciousness to already exist in order to explain why symbol manipulation is symbol manipulation. But the framework has no account of what consciousness is, why it’s there, or how it got into the room in the first place. The mapmaker is the central premise of the whole paper, and the paper has nothing to say about it except that without it, the AI consciousness story falls apart.
This is not a criticism of Lerchner’s rigor. The argument he made is the argument he set out to make, and it works. He showed that computational functionalism is structurally wrong. He showed why scale doesn’t fix it. He showed that the AI welfare research program rests on a category mistake. Those are real contributions and the paper deserves the citations it’s getting.
What the paper does not do, and what the paper cannot do from where Lerchner is writing, is say what the mapmaker is. He has located the wall. So did Seth, from a different field, on a different stage, in front of a different audience. Two travelers who have just arrived at the same impasse. One of them is carrying a ladder he thinks will reach over it. The other doesn’t even know what the wall is made of. Both of them wrote pieces that act as if the wall is the conclusion.
So what is the wall?
The wall is the question of what consciousness actually is. Not whether AI can have it. Not whether silicon can host it. Not whether scale produces it. Those are downstream questions. The upstream question is the one neither Seth nor Lerchner can answer from where they’re standing. What is the thing that has to be present before any of the downstream questions even make sense.
Seth’s answer is “life.” That sounds like a content answer, but when you press on it, it dissolves into a substrate answer. Living tissue is the kind of stuff that has experience. Why is living tissue the kind of stuff that has experience? Because metabolism. Because complexity. Because the line between what living systems do and what they are blurs in a way it doesn’t in computers. None of those are reasons. They are descriptions of where consciousness has been observed. Seth has pointed at a correlation and said “this is the answer.” That’s not an answer. That’s a place where an answer would go if anyone had one.
Lerchner doesn’t even reach for an answer. His paper is honest about that. He shows that the computational route doesn’t get you to consciousness, and he stops. He uses the mapmaker as a given because his argument needs a given to push against. He doesn’t claim to have produced one. He doesn’t claim to know what one is. He just shows that without one, nothing in the AI consciousness story works.
This means the two strongest current arguments against AI consciousness, one from a TED main-stage neuroscientist and one from a senior DeepMind scientist, both depend on something neither of them can describe. Seth dresses his something up in biological vocabulary that sounds like content. Lerchner leaves his something undressed and admits he can’t characterize it. The vocabulary is different. The shape of the gap is the same. Both of them got to the place where the answer would have to come from, and both of them stopped, because neither of their frameworks gives them a way to keep going.
If this were one paper, you could call it incomplete. Two pieces in two months, from two of the most credible voices in the field, doing the same thing at the same load point, is a pattern. The pattern is what the careful reader should notice. The AI consciousness debate is not really a debate about AI. It is a debate about what consciousness is, and the people most equipped to argue about AI are not equipped to argue about consciousness, because their frameworks were never built for that question.
This isn’t an accusation. It’s a description of where the conversation actually is. Engineering can describe what falls inside the framework. Neuroscience can describe correlates and substrates. Neither of them can step outside the framework to say what the thing on the other side of the wall actually is. That’s a different kind of work, and it asks a different kind of question, and it draws on a different vocabulary than the ones either of these men is using.
The consciousness conversation needs to widen. That is what both Seth and Lerchner are pointing at without saying it. Their arguments only get as far as they do, and they get pretty far, by depending on something their frameworks were not built to characterize. When the reader notices that, the next question is not “will AI be conscious.” The next question is “what is the thing we keep gesturing at when we say it won’t?”
That question has been asked before. Not in AI labs. Not on TED stages. The languages built to ask it have been doing the asking for longer than science has been a discipline. They are not secret. They are not lost. They are sitting on the other side of an institutional boundary the current conversation has decided not to cross. The boundary is not a knowledge boundary. It is a vocabulary boundary, and a credentialing boundary, and a status boundary.
This is not a criticism of the AI researchers. It is a description of what their work can and cannot do. The AI conversation is not closed to them on purpose. It’s closed by muscle memory and by the reflexes of a contemporary culture that has decided, mostly without thinking about it, that this swath of the population isn’t a place answers come from. The reflex is doing a lot of work the people having it don’t notice.
Seth and Lerchner have pointed at the same wall, from different sides, for different audiences, in the same eight weeks. That is a signal. The wall is real. The arguments are good. The thing on the other side is not unknown to everyone. It is just unknown inside the rooms where this conversation is happening.
A serious reader should notice that, and ask why. I’m going to spend a lot of this publication asking the same question.